eval

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 11, 2026 License: AGPL-3.0 Imports: 6 Imported by: 0

Documentation

Overview

Package eval provides the evaluation framework for Trace components.

Overview

The eval package is the foundation for proving correctness, measuring performance, and comparing algorithm implementations in Trace. Every component that participates in the CRS (Code Reasoning State) system implements the Evaluable interface, enabling:

  • Property-based correctness verification
  • Performance benchmarking with statistical analysis
  • A/B testing between algorithm variants
  • Chaos injection for resilience testing
  • Regression detection for CI/CD gates

Architecture 

┌─────────────────────────────────────────────────────────────────────────────┐
│                         EVALUATION FRAMEWORK                                 │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                         Registry                                     │    │
│  │  • Stores all Evaluable components                                   │    │
│  │  • Provides lookup by name                                           │    │
│  │  • Manages lifecycle                                                 │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                    │                                         │
│         ┌──────────────────────────┼──────────────────────────┐             │
│         │                          │                          │             │
│         ▼                          ▼                          ▼             │
│  ┌─────────────┐           ┌─────────────┐           ┌─────────────┐        │
│  │ Correctness │           │  Benchmark  │           │  A/B Test   │        │
│  │  Verifier   │           │   Runner    │           │   Harness   │        │
│  └─────────────┘           └─────────────┘           └─────────────┘        │
│         │                          │                          │             │
│         ▼                          ▼                          ▼             │
│  ┌─────────────┐           ┌─────────────┐           ┌─────────────┐        │
│  │  Property   │           │   Metrics   │           │  Statistics │        │
│  │   Tests     │           │   Export    │           │   Analysis  │        │
│  └─────────────┘           └─────────────┘           └─────────────┘        │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

The Evaluable Interface

Every component that can be evaluated implements Evaluable: 

type Evaluable interface {
    Name() string
    Properties() []Property
    Metrics() []MetricDefinition
    HealthCheck(ctx context.Context) error
}

Property-Based Testing

Properties define invariants that must hold for all inputs: 

property := Property{
    Name: "no_soft_signal_clauses",
    Description: "CDCL only learns from compiler/test failures",
    Check: func(input, output any) error {
        // Verify the invariant
    },
    Generator: func() any {
        // Generate random valid input
    },
}

The Verifier runs properties against generated inputs: 

verifier := correctness.NewVerifier(registry)
result, err := verifier.Verify(ctx, "cdcl", correctness.WithIterations(10000))

Usage Example 

// Register a component
registry := eval.NewRegistry()
registry.Register(myAlgorithm)

// Verify correctness
verifier := correctness.NewVerifier(registry)
result, err := verifier.Verify(ctx, "my_algorithm")
if !result.Passed {
    log.Fatalf("Property %s failed: %v", result.FailedProperty, result.FailingInput)
}

Thread Safety

All types in this package are safe for concurrent use unless otherwise documented. The Registry uses read-write locks for concurrent access. Verifiers can run multiple property checks in parallel.

Signal Sources

The eval package enforces the hard/soft signal boundary: 

const (
    SourceCompiler  SignalSource = iota  // Hard signal - can update state
    SourceTest                            // Hard signal - can update state
    SourceTypeCheck                       // Hard signal - can update state
    SourceLinter                          // Hard signal - can update state
    SourceLLM                             // Soft signal - guidance only
    SourceHeuristic                       // Soft signal - guidance only
)

Properties can verify that components respect this boundary.

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrNotFound is returned when a component is not found in the registry.
	ErrNotFound = errors.New("component not found")

	// ErrAlreadyRegistered is returned when attempting to register a duplicate.
	ErrAlreadyRegistered = errors.New("component already registered")

	// ErrNilComponent is returned when attempting to register nil.
	ErrNilComponent = errors.New("component must not be nil")

	// ErrInvalidProperty is returned when a property is malformed.
	ErrInvalidProperty = errors.New("invalid property definition")

	// ErrPropertyFailed is returned when a property check fails.
	ErrPropertyFailed = errors.New("property check failed")

	// ErrHealthCheckFailed is returned when a health check fails.
	ErrHealthCheckFailed = errors.New("health check failed")

	// ErrSoftSignalViolation is returned when soft signals are used for hard decisions.
	ErrSoftSignalViolation = errors.New("soft signal used for state mutation")
)
View Source 
var DefaultRegistry = NewRegistry()

DefaultRegistry is the global registry instance. Components can register themselves during init() using MustRegister.

Functions

func List 

func List() []string

List returns all component names from the default registry.

func MustRegister 

func MustRegister(component Evaluable)

MustRegister registers a component with the default registry, panicking on error.

func Register 

func Register(component Evaluable) error

Register registers a component with the default registry.

Types

type CoverageInfo 

type CoverageInfo struct {
	// Statements is the number of statements covered.
	Statements int

	// TotalStatements is the total number of statements.
	TotalStatements int

	// Branches is the number of branches covered.
	Branches int

	// TotalBranches is the total number of branches.
	TotalBranches int
}

CoverageInfo tracks code coverage during verification.

func (*CoverageInfo) Percentage 

func (c *CoverageInfo) Percentage() float64

Percentage returns the coverage percentage.

type Evaluable 

type Evaluable interface {
	// Name returns a unique identifier for metrics and logging.
	// The name should be stable across versions and suitable for use
	// in metric labels (lowercase, underscore-separated).
	//
	// Example: "cdcl", "pn_mcts", "tms"
	Name() string

	// Properties returns the correctness properties this component guarantees.
	// These are used by the property-based testing framework.
	// An empty slice indicates no properties to verify.
	Properties() []Property

	// Metrics returns the metrics this component exposes.
	// These are used by the benchmark and monitoring systems.
	// An empty slice indicates no custom metrics.
	Metrics() []MetricDefinition

	// HealthCheck verifies the component is functioning correctly.
	// Called during chaos testing recovery verification.
	// Returns nil if healthy, error with details otherwise.
	//
	// Inputs:
	//   - ctx: Context for cancellation. Must not be nil.
	//
	// Outputs:
	//   - error: nil if healthy, descriptive error otherwise.
	HealthCheck(ctx context.Context) error
}

Evaluable is the interface that all testable/benchmarkable components implement. This is the foundation of the evaluation framework.

Thread Safety: Implementations must be safe for concurrent use.

func Get 

func Get(name string) (Evaluable, bool)

Get retrieves a component from the default registry.

type HealthResult 

type HealthResult struct {
	// Component is the name of the component.
	Component string

	// Status is the health status.
	Status HealthStatus

	// Message provides details about the health status.
	Message string

	// Duration is the time spent on the health check.
	Duration time.Duration

	// Timestamp is when the check was performed (Unix milliseconds UTC).
	Timestamp int64

	// Details contains component-specific health information.
	Details map[string]any
}

HealthResult contains the result of a health check.

type HealthStatus 

type HealthStatus int

HealthStatus represents the health state of a component. 

const (
	// HealthUnknown is the zero value.
	HealthUnknown HealthStatus = iota
	// HealthHealthy indicates the component is functioning correctly.
	HealthHealthy
	// HealthDegraded indicates reduced functionality.
	HealthDegraded
	// HealthUnhealthy indicates the component is not functioning.
	HealthUnhealthy
)

func (HealthStatus) String 

func (h HealthStatus) String() string

String returns the string representation of a HealthStatus.

type MetricDefinition 

type MetricDefinition struct {
	// Name is the metric name (should follow Prometheus conventions).
	// Example: "algorithm_process_seconds"
	Name string

	// Type is the metric type (counter, gauge, histogram, summary).
	Type MetricType

	// Description explains what this metric measures.
	Description string

	// Labels are the label names for this metric.
	// Example: []string{"algorithm", "status"}
	Labels []string

	// Buckets are the histogram bucket boundaries (for histograms only).
	// Example: []float64{0.001, 0.01, 0.1, 1.0, 10.0}
	Buckets []float64

	// Objectives are the quantile objectives (for summaries only).
	// Example: map[float64]float64{0.5: 0.05, 0.9: 0.01, 0.99: 0.001}
	Objectives map[float64]float64
}

MetricDefinition describes a metric exposed by a component.

Example: 

MetricDefinition{
    Name:        "cdcl_clauses_learned",
    Type:        MetricCounter,
    Description: "Total number of clauses learned",
    Labels:      []string{"source"},
}

func (*MetricDefinition) Validate 

func (m *MetricDefinition) Validate() error

Validate checks that the metric definition is well-formed.

Outputs:

  • error: nil if valid, descriptive error otherwise.

type MetricType 

type MetricType int

MetricType identifies the type of metric. 

const (
	// MetricCounter is a monotonically increasing value.
	MetricCounter MetricType = iota
	// MetricGauge is a value that can go up or down.
	MetricGauge
	// MetricHistogram records observations in buckets.
	MetricHistogram
	// MetricSummary records observations with quantiles.
	MetricSummary
)

func (MetricType) String 

func (m MetricType) String() string

String returns the string representation of a MetricType.

type Property 

type Property struct {
	// Name is a unique identifier for this property.
	// Should be lowercase with underscores (e.g., "proof_propagation_correct").
	Name string

	// Description explains what this property verifies.
	// Should be a complete sentence.
	Description string

	// Check verifies the property holds for the given input/output pair.
	// Returns nil if the property holds, error with details otherwise.
	//
	// Inputs:
	//   - input: The input that was provided to the component.
	//   - output: The output that was produced.
	//
	// Outputs:
	//   - error: nil if property holds, descriptive error otherwise.
	Check func(input any, output any) error

	// Generator produces random valid inputs for property testing.
	// Should return diverse inputs covering edge cases.
	// If nil, the property can only be checked with explicit inputs.
	Generator func() any

	// Shrink attempts to reduce a failing input to a minimal case.
	// If nil, no shrinking is performed.
	// This helps debugging by finding the smallest failing input.
	Shrink func(input any) []any

	// Tags categorize this property for selective testing.
	// Examples: "critical", "performance", "boundary"
	Tags []string

	// Timeout is the maximum time for a single property check.
	// Zero means use the default timeout.
	Timeout time.Duration
}

Property defines a correctness invariant for testing. Properties should be independent and composable.

Example: 

Property{
    Name: "no_soft_signal_clauses",
    Description: "CDCL only learns from compiler/test failures",
    Check: func(input, output any) error { ... },
    Generator: func() any { ... },
}

func (*Property) HasGenerator 

func (p *Property) HasGenerator() bool

HasGenerator returns true if this property has an input generator.

func (*Property) HasShrink 

func (p *Property) HasShrink() bool

HasShrink returns true if this property supports input shrinking.

func (*Property) HasTag 

func (p *Property) HasTag(tag string) bool

HasTag returns true if this property has the specified tag.

func (*Property) Validate 

func (p *Property) Validate() error

Validate checks that the property is well-formed.

Outputs:

  • error: nil if valid, descriptive error otherwise.

type PropertyResult 

type PropertyResult struct {
	// Name is the property name.
	Name string

	// Passed is true if the property held for all inputs.
	Passed bool

	// Iterations is the number of test iterations run.
	Iterations int

	// Duration is the time spent on this property.
	Duration time.Duration

	// FailingInput is the input that caused failure (if any).
	// This is the minimal input after shrinking.
	FailingInput any

	// FailingOutput is the output that caused failure (if any).
	FailingOutput any

	// Error is the error returned by the Check function (if any).
	Error error

	// ShrinkSteps is the number of shrinking iterations performed.
	ShrinkSteps int
}

PropertyResult contains the result of verifying a single property.

type RegistrationHook 

type RegistrationHook func(name string, component Evaluable, registered bool)

RegistrationHook is called when a component is registered or unregistered.

type Registry 

type Registry struct {
	// contains filtered or unexported fields
}

Registry manages all evaluable components in the system.

Description: 

The Registry provides a central location for registering and looking up
evaluable components. It supports concurrent access and provides methods
for batch operations like health checks.

Thread Safety: Safe for concurrent use via read-write mutex.

func NewRegistry 

func NewRegistry() *Registry

NewRegistry creates a new empty registry.

Outputs:

  • *Registry: The new registry. Never nil.

Example: 

registry := eval.NewRegistry()
registry.Register(myAlgorithm)

func (*Registry) AddHook 

func (r *Registry) AddHook(hook RegistrationHook)

AddHook adds a registration hook.

Description: 

Hooks are called when components are registered or unregistered.
They receive the component name, the component, and a boolean
indicating whether it was registered (true) or unregistered (false).

Inputs:

  • hook: The hook function to add.

Thread Safety: Safe for concurrent use.

func (*Registry) All 

func (r *Registry) All() map[string]Evaluable

All returns all registered components.

Outputs:

  • map[string]Evaluable: Copy of the components map.

Thread Safety: Safe for concurrent use.

func (*Registry) Clear 

func (r *Registry) Clear()

Clear removes all components from the registry.

Thread Safety: Safe for concurrent use.

func (*Registry) Count 

func (r *Registry) Count() int

Count returns the number of registered components.

Outputs:

  • int: Number of components.

Thread Safety: Safe for concurrent use.

func (*Registry) FindByTag 

func (r *Registry) FindByTag(tag string) []string

FindByTag finds all components that have a property with the given tag.

Inputs:

  • tag: The tag to search for.

Outputs:

  • []string: Names of components with properties having the tag.

Thread Safety: Safe for concurrent use.

func (*Registry) Get 

func (r *Registry) Get(name string) (Evaluable, bool)

Get retrieves a component by name.

Inputs:

  • name: The name of the component to retrieve.

Outputs:

  • Evaluable: The component, or nil if not found.
  • bool: true if found, false otherwise.

Thread Safety: Safe for concurrent use.

Example: 

if component, ok := registry.Get("cdcl"); ok {
    // Use component
}

func (*Registry) GetAllMetrics 

func (r *Registry) GetAllMetrics() map[string][]MetricDefinition

GetAllMetrics returns all metric definitions from all registered components.

Outputs:

  • map[string][]MetricDefinition: Map from component name to its metrics.

Thread Safety: Safe for concurrent use.

func (*Registry) GetAllProperties 

func (r *Registry) GetAllProperties() map[string][]Property

GetAllProperties returns all properties from all registered components.

Outputs:

  • map[string][]Property: Map from component name to its properties.

Thread Safety: Safe for concurrent use.

func (*Registry) HealthCheckAll 

func (r *Registry) HealthCheckAll(ctx context.Context, concurrency int) []HealthResult

HealthCheckAll runs health checks on all registered components.

Description: 

Runs health checks concurrently with the given concurrency limit.
Returns results for all components, including those that fail.

Inputs:

  • ctx: Context for cancellation. Must not be nil.
  • concurrency: Maximum number of concurrent health checks. If <= 0, defaults to 10.

Outputs:

  • []HealthResult: Results for all components.

Thread Safety: Safe for concurrent use.

Example: 

results := registry.HealthCheckAll(ctx, 5)
for _, result := range results {
    if result.Status != eval.HealthHealthy {
        log.Printf("Unhealthy: %s - %s", result.Component, result.Message)
    }
}

func (*Registry) List 

func (r *Registry) List() []string

List returns all registered component names.

Outputs:

  • []string: Sorted list of component names.

Thread Safety: Safe for concurrent use.

func (*Registry) MustGet 

func (r *Registry) MustGet(name string) Evaluable

MustGet retrieves a component by name, panicking if not found.

Inputs:

  • name: The name of the component to retrieve.

Outputs:

  • Evaluable: The component. Panics if not found.

Thread Safety: Safe for concurrent use.

func (*Registry) MustRegister 

func (r *Registry) MustRegister(component Evaluable)

MustRegister registers a component and panics on error.

Description: 

Convenience method for registration during initialization.
Should only be used during startup, not at runtime.

Inputs:

  • component: The evaluable component to register. Must not be nil.

Thread Safety: Safe for concurrent use.

Example: 

func init() {
    DefaultRegistry.MustRegister(myAlgorithm)
}

func (*Registry) Register 

func (r *Registry) Register(component Evaluable) error

Register adds a component to the registry.

Description: 

Registers the component under its Name(). The name must be unique
within the registry.

Inputs:

  • component: The evaluable component to register. Must not be nil.

Outputs:

  • error: nil on success, ErrNilComponent if component is nil, ErrAlreadyRegistered if name is already taken.

Thread Safety: Safe for concurrent use.

Example: 

err := registry.Register(myAlgorithm)
if err != nil {
    log.Fatalf("Failed to register: %v", err)
}

func (*Registry) Unregister 

func (r *Registry) Unregister(name string) error

Unregister removes a component from the registry.

Description: 

Removes the component with the given name. Returns an error if
the component is not found.

Inputs:

  • name: The name of the component to unregister.

Outputs:

  • error: nil on success, ErrNotFound if not registered.

Thread Safety: Safe for concurrent use.

type SignalSource 

type SignalSource int

SignalSource identifies where a piece of information came from. This is critical for the hard/soft signal boundary. 

const (
	// SourceUnknown is the zero value, indicating unset source.
	SourceUnknown SignalSource = iota

	// Hard signals - can update reasoning state
	// SourceCompiler indicates the signal came from compiler output.
	SourceCompiler
	// SourceTest indicates the signal came from test execution.
	SourceTest
	// SourceTypeCheck indicates the signal came from type checker.
	SourceTypeCheck
	// SourceLinter indicates the signal came from linter output.
	SourceLinter
	// SourceSyntax indicates the signal came from syntax analysis.
	SourceSyntax

	// Soft signals - guidance only, cannot update state
	// SourceLLM indicates the signal came from LLM output.
	SourceLLM
	// SourceHeuristic indicates the signal came from heuristic estimation.
	SourceHeuristic
	// SourceSimilarity indicates the signal came from similarity matching.
	SourceSimilarity
	// SourceEstimate indicates the signal came from statistical estimation.
	SourceEstimate
)

func (SignalSource) IsHard 

func (s SignalSource) IsHard() bool

IsHard returns true if this is a hard signal that can update state.

func (SignalSource) IsSoft 

func (s SignalSource) IsSoft() bool

IsSoft returns true if this is a soft signal (guidance only).

func (SignalSource) String 

func (s SignalSource) String() string

String returns the string representation of a SignalSource.

type SimpleEvaluable 

type SimpleEvaluable struct {
	// contains filtered or unexported fields
}

SimpleEvaluable is a simple implementation of Evaluable for testing.

func NewSimpleEvaluable 

func NewSimpleEvaluable(name string) *SimpleEvaluable

NewSimpleEvaluable creates a new SimpleEvaluable.

Inputs:

  • name: Unique identifier for the component. Must not be empty.

Outputs:

  • *SimpleEvaluable: The new evaluable. Never nil.

func (*SimpleEvaluable) AddMetric 

func (s *SimpleEvaluable) AddMetric(m MetricDefinition) *SimpleEvaluable

AddMetric adds a metric definition to this evaluable.

func (*SimpleEvaluable) AddProperty 

func (s *SimpleEvaluable) AddProperty(p Property) *SimpleEvaluable

AddProperty adds a property to this evaluable.

func (*SimpleEvaluable) HealthCheck 

func (s *SimpleEvaluable) HealthCheck(ctx context.Context) error

HealthCheck performs the health check.

func (*SimpleEvaluable) Metrics 

func (s *SimpleEvaluable) Metrics() []MetricDefinition

Metrics returns the registered metrics.

func (*SimpleEvaluable) Name 

func (s *SimpleEvaluable) Name() string

Name returns the component name.

func (*SimpleEvaluable) Properties 

func (s *SimpleEvaluable) Properties() []Property

Properties returns the registered properties.

func (*SimpleEvaluable) SetHealthCheck 

func (s *SimpleEvaluable) SetHealthCheck(fn func(ctx context.Context) error) *SimpleEvaluable

SetHealthCheck sets the health check function.

type VerifyResult 

type VerifyResult struct {
	// Component is the name of the component that was verified.
	Component string

	// Properties contains results for each property.
	Properties []PropertyResult

	// Duration is the total time spent verifying.
	Duration time.Duration

	// Passed is true if all properties passed.
	Passed bool

	// Iterations is the total number of test iterations run.
	Iterations int

	// Coverage tracks which code paths were exercised.
	Coverage *CoverageInfo
}

VerifyResult contains the results of verifying a component's properties.

func (*VerifyResult) FailedProperties 

func (r *VerifyResult) FailedProperties() []PropertyResult

FailedProperties returns the properties that failed.

Source Files

View all Source files

Directories

Path Synopsis
Package ab provides A/B testing harness for comparing algorithm implementations.
 Package ab provides A/B testing harness for comparing algorithm implementations.
Package benchmark provides performance benchmarking for evaluable components.
 Package benchmark provides performance benchmarking for evaluable components.
Package chaos provides fault injection for resilience testing.
 Package chaos provides fault injection for resilience testing.
Package regression provides CI/CD regression detection gates.
 Package regression provides CI/CD regression detection gates.
Package telemetry provides observability infrastructure for the evaluation framework.
 Package telemetry provides observability infrastructure for the evaluation framework.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.