eval

package
v0.1.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 11, 2026 License: MIT Imports: 6 Imported by: 0

Documentation

Overview

Package eval provides builtin evaluation suites.

Package eval provides evaluation framework for hawk-eco. It supports standardized test suites, agent evals, and scoring.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func GetFixture

func GetFixture(name string) (string, bool)

GetFixture retrieves a fixture by name.

func Materialize

func Materialize(test *Test) string

Materialize builds a command with fixtures substituted.

func Now

func Now() int64

Now returns current time in milliseconds.

func RunCommand

func RunCommand(cmd string) (string, error)

RunCommand runs a command and returns output.

func RunShellCommand

func RunShellCommand(cmd string) (string, error)

RunShellCommand runs a shell command.

Types

type ExpectedResult

type ExpectedResult struct {
	ExitCode int
	Output   string
	Error    string
}

ExpectedResult represents expected test output.

type Report

type Report struct {
	SuiteName string
	Results   []*Result
	Summary   *Summary
}

Report represents evaluation report.

type Result

type Result struct {
	Test       *Test
	Passed     bool
	Output     string
	Error      error
	DurationMs int64
}

Result represents the result of running an eval test.

type Suite

type Suite struct {
	Name        string
	Description string
	Fixtures    []string
	Tests       []*Test
}

Suite represents an evaluation suite with fixtures and tests.

func BuiltinSuites

func BuiltinSuites() []Suite

BuiltinSuites returns default evaluation suites.

type SuiteManager

type SuiteManager struct {
	// contains filtered or unexported fields
}

SuiteManager manages evaluation suites.

func NewSuiteManager

func NewSuiteManager() *SuiteManager

NewSuiteManager creates a new suite manager.

func (*SuiteManager) FindSuite

func (m *SuiteManager) FindSuite(name string) *Suite

FindSuite finds a suite by name.

func (*SuiteManager) List

func (m *SuiteManager) List() []*Suite

List returns all registered suites.

func (*SuiteManager) LoadDefaultSuites

func (m *SuiteManager) LoadDefaultSuites() error

LoadDefaultSuites loads suites from default locations.

func (*SuiteManager) LoadFixture

func (m *SuiteManager) LoadFixture(name string) (string, bool)

LoadFixture loads a fixture file.

func (*SuiteManager) LoadSuitesFromDir

func (m *SuiteManager) LoadSuitesFromDir(dir string) error

LoadSuitesFromDir loads suites from a directory.

func (*SuiteManager) Register

func (m *SuiteManager) Register(suite *Suite)

Register adds a suite to the manager.

func (*SuiteManager) RegisterFixture

func (m *SuiteManager) RegisterFixture(name, content string)

RegisterFixture registers a fixture.

func (*SuiteManager) RunAllSuites

func (m *SuiteManager) RunAllSuites() []*Report

RunAllSuites runs all registered suites.

func (*SuiteManager) RunSuite

func (m *SuiteManager) RunSuite(name string) *Report

RunSuite runs all tests in a suite.

func (*SuiteManager) RunTest

func (m *SuiteManager) RunTest(test *Test) *Result

RunTest runs a single test.

func (*SuiteManager) Summarize

func (m *SuiteManager) Summarize(results []*Result, totalDuration int64) *Summary

Summarize creates a summary from results.

type Summary

type Summary struct {
	TotalTests  int
	PassedTests int
	FailedTests int
	PassRate    float64
	TotalTimeMs int64
}

Summary represents evaluation summary.

type Test

type Test struct {
	Name        string
	Description string
	Command     string
	Args        []string
	Expected    ExpectedResult
}

Test represents a single evaluation test.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL