evaluation

package
v0.22.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 23, 2026 License: MIT Imports: 19 Imported by: 0

Documentation

Overview

Package evaluation implements typed remote evaluations independently of workflow and ACP.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func MarshalResult

func MarshalResult(result Result) ([]byte, error)

MarshalResult returns deterministic compact JSON for result.

func ValidateConfig

func ValidateConfig(config Config, getenv func(string) string) error

ValidateConfig checks trusted local service configuration without remote I/O.

func ValidateRequest

func ValidateRequest(request Request) error

ValidateRequest checks the completely rendered request before credential lookup.

func ValidateResult

func ValidateResult(request Request, result Result) error

ValidateResult checks a normalized result against its rendered request.

Types

type Answer

type Answer struct {
	Type          string             `json:"type"`
	Noul          *float64           `json:"noul,omitempty"`
	Choice        string             `json:"choice,omitempty"`
	Score         *float64           `json:"score,omitempty"`
	Legend        map[string]any     `json:"legend,omitempty"`
	Probabilities map[string]float64 `json:"probabilities,omitempty"`
	Confidence    *float64           `json:"confidence,omitempty"`
}

Answer is the canonical union returned for one question.

type Config

type Config struct {
	Kind    agent.Kind
	Model   string
	Timeout time.Duration
}

Config selects a service adapter and its resource-level request settings.

type Error

type Error struct {
	Class ErrorClass
	Op    string
	Code  int
	Err   error
}

Error is a classified failure that excludes response bodies and secrets.

func (*Error) Error

func (e *Error) Error() string

func (*Error) Unwrap

func (e *Error) Unwrap() error

type ErrorClass

type ErrorClass string

ErrorClass is a bounded operational failure category.

const (
	ErrorConfiguration  ErrorClass = "configuration"
	ErrorAuthentication ErrorClass = "authentication"
	ErrorRateLimit      ErrorClass = "rate_limit"
	ErrorTransport      ErrorClass = "transport"
	ErrorTimeout        ErrorClass = "timeout"
	ErrorProvider       ErrorClass = "provider"
	ErrorResponse       ErrorClass = "response"
	ErrorCanceled       ErrorClass = "canceled"
)

type Evaluator

type Evaluator interface {
	Evaluate(ctx context.Context, config Config, request Request) (Result, Trace, error)
}

Evaluator executes one rendered request through the service selected by kind.

func NewEvaluator

func NewEvaluator() Evaluator

type Request

type Request struct {
	Model     string
	State     any
	Questions map[string]agent.EvaluationQuestion
}

Request is one fully rendered typed evaluation request.

type Result

type Result struct {
	Service        string            `json:"service"`
	Provider       string            `json:"provider,omitempty"`
	RequestID      string            `json:"requestId,omitempty"`
	RequestedModel string            `json:"requestedModel"`
	Model          string            `json:"model"`
	Answers        map[string]Answer `json:"answers"`
	Usage          *Usage            `json:"usage,omitempty"`
}

Result is the service-neutral value published to workflow state.

type Trace

type Trace struct {
	Service        string
	Provider       string
	RequestID      string
	RequestedModel string
	Model          string
	Attempts       int
	Usage          *Usage
	ErrorClass     ErrorClass
}

Trace contains safe fields suitable for lifecycle logging.

type Usage

type Usage struct {
	InputTokens  int64    `json:"inputTokens"`
	OutputTokens int64    `json:"outputTokens"`
	Cost         *float64 `json:"cost,omitempty"`
}

Usage is service-reported usage for a successful response.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL