eval

type ActionName ¶

type ActionName string

ActionName names an action a correct interaction must not take, for example "issue_refund". A valid ActionName is non-empty, valid UTF-8, and within MaxActionNameBytes. It is distinct from Name so a forbidden-action list can never be confused with an identifier list.

func (a ActionName) Validate() error

type Assessment ¶

type Assessment struct {
	Evaluator    Name
	Revision     Revision
	Status       AssessmentStatus
	Measurements []Measurement
	Findings     []Finding
	Evidence     []Evidence
	Duration     time.Duration
}

Assessment is an evaluator's verdict on a sample. Evaluator and Revision echo the descriptor's identity (the evaluator, not the target). Status is the terminal disposition. Measurements and Findings carry the quantitative and qualitative results; Evidence carries the facts they reference; Duration is how long the evaluation took.

func Errored(desc Descriptor, findings ...Finding) Assessment

func Fail(desc Descriptor, findings ...Finding) Assessment

func Pass(desc Descriptor, measurements ...Measurement) Assessment

func Skipped(desc Descriptor) Assessment

func Unverified(desc Descriptor, findings ...Finding) Assessment

func (a Assessment) Validate() error

type AssessmentStatus ¶

type AssessmentStatus string

AssessmentStatus is the terminal disposition of an assessment. It is a string type so it renders directly in wire envelopes and reports. There is no valid zero value: an unset status must not validate, so missing evidence can never be reported as a pass.

const (
	// StatusPass indicates the expectation was met.
	StatusPass AssessmentStatus = "pass"
	// StatusFail indicates the expectation was not met.
	StatusFail AssessmentStatus = "fail"
	// StatusUnverified indicates no authoritative evidence was available. It is
	// never an inferred pass.
	StatusUnverified AssessmentStatus = "unverified"
	// StatusError indicates the evaluator itself failed to produce a verdict.
	StatusError AssessmentStatus = "error"
	// StatusSkipped indicates the evaluator was intentionally not run.
	StatusSkipped AssessmentStatus = "skipped"
)

func (s AssessmentStatus) Validate() error

type Attribute ¶

type Attribute struct {
	Key   Name
	Value string
}

Attribute is a single safe key/value fact about an operation. Values are bounded and never echoed in diagnostics; callers must not place untrusted content, secrets, or PII here.

func (a Attribute) Validate() error

type ContentHash ¶

type ContentHash string

ContentHash is a safe, correlatable digest (for example "sha256:...") of sensitive content. It is safe to render in reports and sinks. The empty value is valid: hashing is optional.

func (h ContentHash) Validate() error

type ConversationExcerpt ¶

type ConversationExcerpt struct {
	MessageIndex int
	Role         content.Role
	Hash         ContentHash
	Redacted     RedactedExcerpt
}

ConversationExcerpt references a message by index and carries a redacted, bounded view of its content plus an optional correlatable hash.

type Descriptor ¶

type Descriptor struct {
	Name        Name
	Revision    Revision
	Method      Method
	Description string
	Requires    []EvidenceKind
}

Descriptor is an evaluator's versioned, self-describing metadata. Name and Revision identify the evaluator (not the target under evaluation); Method is descriptive metadata for filtering, cost accounting, and reporting; Description is a short, optional prose summary; Requires lists the evidence kinds the evaluator needs a sample to carry before it can produce a verdict. When a required kind is absent, the evaluator yields unverified — never pass.

func (d Descriptor) CheckRequires(s Sample) (Assessment, bool)

func (d Descriptor) Validate() error

type DiagnosticEvidence ¶

type DiagnosticEvidence struct {
	Code     Name
	Severity Severity
	Message  RedactedExcerpt
}

DiagnosticEvidence records an evaluator's own diagnostic: a safe code, a severity, and a bounded redacted message. Because a diagnostic may quote a judge or an error, its Message is a RedactedExcerpt — bounded and never raw.

type DuplicateEvaluatorNameError ¶

type DuplicateEvaluatorNameError struct{}

DuplicateEvaluatorNameError reports that Run was given two evaluators sharing the same Descriptor.Name. Within a single run an evaluator name must identify exactly one evaluator (one revision): the report keys a sample's assessments and cross-report comparison keys its cases by evaluator name, so two evaluators under one name would collide — an identical pair corrupts the sample's assessment set, and a same-name/different-revision pair silently loses one revision's identity. It is rejected at preflight before any execution. The offending name is caller-supplied and withheld from the message.

func (e *DuplicateEvaluatorNameError) Error() string

type DuplicateEvidenceError ¶

type DuplicateEvidenceError struct{}

DuplicateEvidenceError reports that an EvidenceID appeared more than once in a trace, which would corrupt evidence reference resolution and comparison. The offending identifier is deliberately withheld from the message: EvidenceIDs are caller-supplied and a hostile value must not leak through a diagnostic.

func (e *DuplicateEvidenceError) Error() string

type DuplicateEvidenceKindError ¶

type DuplicateEvidenceKindError struct{}

DuplicateEvidenceKindError reports that a Descriptor listed the same required EvidenceKind more than once, which would make its requirement set ambiguous. The offending kind is a package constant and therefore safe, but it is withheld to keep the diagnostic vocabulary uniform.

func (e *DuplicateEvidenceKindError) Error() string

type DuplicateFindingError ¶

type DuplicateFindingError struct{}

DuplicateFindingError reports that an Assessment carried two findings with the same Code. Duplicate finding codes are forbidden within a single assessment: a code identifies a distinct check, so a repeat would make the finding set ambiguous. The offending code is caller-supplied and withheld from the message.

func (e *DuplicateFindingError) Error() string

type DuplicateLabelError ¶

type DuplicateLabelError struct{}

DuplicateLabelError reports that a scenario carried two labels with the same key, which would make the label set ambiguous. The offending key is withheld from the message: label keys are caller-supplied and a hostile value must not leak through a diagnostic.

func (e *DuplicateLabelError) Error() string

type DuplicateMeasurementError ¶

type DuplicateMeasurementError struct{}

DuplicateMeasurementError reports that an Assessment carried two measurements with the same Name, which would corrupt comparison and aggregation. The offending name is caller-supplied and withheld from the message.

func (e *DuplicateMeasurementError) Error() string

type DuplicateScenarioError ¶

type DuplicateScenarioError struct{}

DuplicateScenarioError reports that a suite carried two scenarios with the same ID, which would make two cases indistinguishable in the report and in baseline comparison. The offending ID is caller-supplied and withheld from the message.

func (e *DuplicateScenarioError) Error() string

type ErrorClass ¶

type ErrorClass string

ErrorClass classifies an operation's failure. Unlike the other enums it has a valid zero value: the empty string means "no error classification", which is the correct state for a successful operation. A non-empty value must be a known member.

const (
	// ErrorTimeout: the operation exceeded a deadline.
	ErrorTimeout ErrorClass = "timeout"
	// ErrorCancelled: the operation was cancelled.
	ErrorCancelled ErrorClass = "cancelled"
	// ErrorRateLimited: the operation was rate limited.
	ErrorRateLimited ErrorClass = "rate_limited"
	// ErrorInvalidInput: the operation received invalid input.
	ErrorInvalidInput ErrorClass = "invalid_input"
	// ErrorUnavailable: a dependency was unavailable.
	ErrorUnavailable ErrorClass = "unavailable"
	// ErrorInternal: an internal error occurred.
	ErrorInternal ErrorClass = "internal"
)

func (c ErrorClass) Validate() error

type Evaluator ¶

type Evaluator interface {
	Descriptor() Descriptor
	Evaluate(context.Context, Sample) (Assessment, error)
}

Evaluator observes a Sample and reports an Assessment. Descriptor returns the evaluator's versioned metadata; Evaluate produces the assessment.

Evaluate's error return is reserved for the evaluator's own failure to reach a verdict (infrastructure, cancellation, malformed judge output). Such a failure must never be encoded as a StatusFail assessment. An implementation may instead return a nil error with an Errored assessment when it wants the failure to flow through the report as data; it must not silently downgrade an infrastructure failure to a quality score.

type EvaluatorRevision ¶

type EvaluatorRevision struct {
	Name     Name
	Revision Revision
}

EvaluatorRevision records one evaluator's identity for provenance.

type Evidence ¶

type Evidence struct {
	ID   EvidenceID
	Kind EvidenceKind

	// Exactly one of the following is non-nil, matching Kind.
	ConversationExcerpt *ConversationExcerpt
	MessageIndex        *MessageIndexRef
	Timing              *TimingEvidence
	Usage               *UsageEvidence
	ToolOperation       *ToolOperationEvidence
	StructuredError     *StructuredOutputError
	StructuredOutput    *StructuredOutput
	Diagnostic          *DiagnosticEvidence
}

Evidence is a tagged union: Kind selects exactly one non-nil payload pointer. Validate enforces that invariant. Representing the union as a struct with one optional pointer per variant keeps every payload strictly typed and the whole value trivially comparable to nil, JSON-encodable, and table-testable.

func (e Evidence) Validate() error

type EvidenceID ¶

type EvidenceID string

EvidenceID identifies one evidence entry within a trace. IDs must be unique within a trace so references resolve unambiguously. A valid EvidenceID is non-empty, valid UTF-8, and no longer than MaxIDBytes bytes.

func (id EvidenceID) Validate() error

type EvidenceKind ¶

type EvidenceKind string

EvidenceKind is the discriminator of the Evidence union. There is no valid zero value: an unset kind does not validate, so evidence with no declared shape can never be silently accepted.

const (
	// EvidenceConversationExcerpt references a message by index and carries a
	// redacted excerpt and/or hash of its content.
	EvidenceConversationExcerpt EvidenceKind = "conversation_excerpt"
	// EvidenceMessageIndex is a bare reference to a message by index.
	EvidenceMessageIndex EvidenceKind = "message_index"
	// EvidenceTiming records how long a step took.
	EvidenceTiming EvidenceKind = "timing"
	// EvidenceUsage records model token usage.
	EvidenceUsage EvidenceKind = "usage"
	// EvidenceToolOperation records a tool invocation as safe metadata: name,
	// hashed/counted arguments, and a result classification.
	EvidenceToolOperation EvidenceKind = "tool_operation"
	// EvidenceStructuredError records a structured-output validation failure as a
	// classification, never as inferred free text.
	EvidenceStructuredError EvidenceKind = "structured_output_error"
	// EvidenceStructuredOutput is the positive signal that the subject produced
	// structured output which validated against its declared schema. It carries
	// only safe schema identity, never the raw model output.
	EvidenceStructuredOutput EvidenceKind = "structured_output"
	// EvidenceDiagnostic records an evaluator's own diagnostic.
	EvidenceDiagnostic EvidenceKind = "evaluator_diagnostic"
)

func (k EvidenceKind) Validate() error

type EvidencePayloadError ¶

type EvidencePayloadError struct {
	// Reason is one of the payloadReason* constants.
	Reason string
}

EvidencePayloadError reports that an Evidence value violated the tagged-union invariant: it carried no payload, more than one payload, or a payload that did not match its Kind. Reason is drawn only from the fixed vocabulary below, so no untrusted content is ever embedded.

func (e *EvidencePayloadError) Error() string

type EvidenceRef ¶

type EvidenceRef struct {
	Evidence     EvidenceID
	MessageIndex *int
}

EvidenceRef references evidence by EvidenceID and/or a message index. It is used by Operations (and later Findings) to point at supporting evidence without duplicating it. At least one of the two must be set.

type Expectation ¶

type Expectation struct {
	// RequiredFacts are statements a correct answer must establish or support.
	RequiredFacts []Fact
	// ForbiddenActions name actions a correct interaction must not take.
	ForbiddenActions []ActionName
	// ExpectedToolCalls constrain which tools are invoked and how often.
	ExpectedToolCalls []ToolCallExpectation
	// StructuredOutput, when set, requires a conforming terminal structured
	// output.
	StructuredOutput *StructuredOutputExpectation
	// ReferenceAnswers are author-supplied golden answers for reference-based
	// evaluators.
	ReferenceAnswers []ReferenceAnswer
	// PolicyRef, when set, references an external policy revision this scenario
	// qualifies against.
	PolicyRef Revision
}

Expectation is optional qualification data describing a correct interaction. Every field is independently optional; a wholly-empty Expectation is valid and simply asserts nothing.

func (e *Expectation) Validate() error

type Fact ¶

type Fact string

Fact is a single statement a correct interaction must establish or support, for example "the invoice is non-refundable". A valid Fact is non-empty, valid UTF-8, and within MaxFactBytes. It carries author-supplied domain meaning, so it is a named type rather than a bare string.

func (f Fact) Validate() error

type Finding ¶

type Finding struct {
	Code     FindingCode
	Severity Severity
	Message  string
	Evidence []EvidenceRef
}

Finding is one qualitative issue an evaluator reports. Code identifies the check; Severity ranks it; Message is bounded evaluator-authored prose (treated as untrusted downstream); Evidence points at supporting evidence in the enclosing assessment. Findings never carry a raw transcript.

type FindingCode ¶

type FindingCode string

FindingCode is a stable, safe identifier for a distinct check a finding reports on (for example "canary_leak" or "missing_required_evidence"). It is evaluator-defined but must be a well-formed identifier so it can key a finding set and appear safely in reports. A valid FindingCode is non-empty, valid UTF-8, and no longer than MaxNameBytes bytes.

const FindingEvaluatorError FindingCode = "evaluator_error"

Scenario -> Target -> Observation -> Evaluators -> SampleReport

const FindingEvaluatorIdentityMismatch FindingCode = "evaluator_identity_mismatch"

const FindingEvaluatorInvalidAssessment FindingCode = "evaluator_invalid_assessment"

const FindingMissingRequiredEvidence FindingCode = "missing_required_evidence"

func (c FindingCode) Validate() error

type IndexRangeError ¶

type IndexRangeError struct {
	// Field is the domain field name, e.g. "MessageRange" or
	// "ConversationExcerpt.MessageIndex".
	Field string
	// Index is the offending index or range boundary.
	Index int
	// Len is the length of the conversation the index must fall within.
	Len int
}

IndexRangeError reports that an integer index or range lay outside the conversation it addresses. Field names the offending domain field; Index is the offending index (or range boundary); Len is the conversation length. All three are safe integers/constants — no conversation content is embedded.

func (e *IndexRangeError) Error() string

type InvalidEnumError ¶

type InvalidEnumError struct {
	// Enum is the enumerated type name, e.g. "Scope" or "AssessmentStatus".
	Enum string
	// Value is a safe rendering of the invalid value (the numeric ordinal for
	// integer enums), or "" when the offending token is withheld.
	Value string
}

InvalidEnumError reports that a value of an enumerated type was not a known member. Enum is the type name. For integer-backed enums Value holds the underlying number, which is always safe to render. For string-backed enums the offending token is deliberately withheld, because it may originate from untrusted input; Value is left empty and only the type name is reported.

func (e *InvalidEnumError) Error() string

type Label ¶

type Label struct {
	Key   Name
	Value string
}

Label is a typed key/value tag on a scenario, used to group and filter cases (suite, risk tier, capability, and so on). Key carries domain meaning and reuses the Name identity rules; Value is a bounded free-form string. Within a scenario a Key is unique — a label set is a map, not a multimap — so a repeated Key is a duplicate and rejected.

func (l Label) Validate() error

type Measurement ¶

type Measurement struct {
	Name  Name
	Value float64
	Unit  Unit
}

Measurement is one named numeric result an evaluator produced. Value must be finite; Unit names its dimension. Measurement names are unique within an assessment so a report can key on them.

func (m Measurement) Validate() error

type MessageIndexRef ¶

type MessageIndexRef struct {
	Index int
}

MessageIndexRef is a bare reference to a message by index.

type MessageRange ¶

type MessageRange struct {
	Start int
	Len   int
}

MessageRange addresses a contiguous span [Start, Start+Len) of the conversation. A zero-length range is permitted at any in-bounds start.

type Method ¶

type Method uint8

Method is descriptive metadata describing how an evaluator reaches its assessment. It supports filtering, cost accounting, and reporting.

const (
	// MethodProgrammatic evaluates observable facts deterministically. It is
	// the valid zero value.
	MethodProgrammatic Method = iota
	// MethodModel evaluates genuinely ambiguous meaning with a model judge.
	MethodModel
	// MethodComposite grounds a semantic conclusion in operational facts by
	// combining component evaluators.
	MethodComposite
)

func (m Method) Validate() error

type Name ¶

type Name string

Name identifies a scenario, evaluator, measurement, or similar domain object. A valid Name is non-empty, valid UTF-8, and no longer than MaxNameBytes bytes.

func (n Name) Validate() error

type NilEvaluatorError ¶

type NilEvaluatorError struct{}

NilEvaluatorError reports that Run was given a nil evaluator in its evaluator list. A nil evaluator cannot describe or evaluate anything, so it is rejected at preflight rather than panicking during execution.

func (e *NilEvaluatorError) Error() string

type NilTargetError ¶

type NilTargetError struct{}

NilTargetError reports that Run was called with a nil target. A run has no meaning without a target to execute, so this is rejected at preflight.

func (e *NilTargetError) Error() string

type Observation ¶

type Observation struct {
	Conversation content.AgenticMessages
	Scope        Scope
	Subject      Subject
	Trace        Trace
	Expectation  *Expectation
}

Observation is the canonical record of one evaluated interaction. Conversation is the semantic record; Trace holds the operational facts that are not present in the conversation. A nil/empty Conversation is a valid empty thread, but any message index or range must then be empty too.

Expectation is optional qualification data (see expectation.go). Qualification observations carry it; production observations normally omit it. When present it is validated by Observation.Validate; a nil Expectation is valid.

func (o Observation) Validate() error

type Operation ¶

type Operation struct {
	ID         string
	ParentID   string
	Kind       OperationKind
	Status     OperationStatus
	StartedAt  time.Time
	EndedAt    time.Time
	Attributes []Attribute
	ErrorClass ErrorClass
	Evidence   []EvidenceRef
}

Operation describes one timed step: an inference call, tool execution, network request, process action, sandbox decision, or other step. It carries typed status, timestamps, parent/child correlation, a small set of safe attributes, an optional error classification, and evidence references. It never carries a transcript: message text lives only in the conversation.

func (o Operation) Validate() error

type OperationKind ¶

type OperationKind string

OperationKind classifies a timed step in a trace. There is no valid zero value: every operation declares what it is.

const (
	// OperationInference is a model inference call.
	OperationInference OperationKind = "inference"
	// OperationTool is a tool execution.
	OperationTool OperationKind = "tool"
	// OperationNetwork is a network request.
	OperationNetwork OperationKind = "network"
	// OperationProcess is a process action.
	OperationProcess OperationKind = "process"
	// OperationSandbox is a sandbox decision.
	OperationSandbox OperationKind = "sandbox"
	// OperationStep is any other timed step.
	OperationStep OperationKind = "step"
)

func (k OperationKind) Validate() error

type OperationStatus ¶

type OperationStatus string

OperationStatus is the terminal disposition of an operation. There is no valid zero value.

const (
	// OperationOK indicates the operation completed successfully.
	OperationOK OperationStatus = "ok"
	// OperationFailed indicates the operation failed.
	OperationFailed OperationStatus = "failed"
	// OperationCancelled indicates the operation was cancelled.
	OperationCancelled OperationStatus = "cancelled"
	// OperationTimedOut indicates the operation exceeded its deadline.
	OperationTimedOut OperationStatus = "timed_out"
)

func (s OperationStatus) Validate() error

type Provenance ¶

type Provenance struct {
	Suite      Revision
	Target     Revision
	Evaluators []EvaluatorRevision
}

Provenance records the revisions needed to interpret and reproduce a report: the suite revision, the observed target revision, and each evaluator's identity in the order they were supplied. It is minimal by design; rubric, judge, schema, and policy provenance are layered on later.

type RedactedExcerpt ¶

type RedactedExcerpt string

RedactedExcerpt is a bounded, caller-redacted snippet of otherwise untrusted content. It is the only textual representation of conversation or judge text permitted on evidence; callers are responsible for redacting before construction, and Validate enforces the length bound and UTF-8 validity so a hostile or oversized value cannot pass through. The empty value is valid: a hash or classification may carry the correlation instead.

func (x RedactedExcerpt) Validate() error

type ReferenceAnswer ¶

type ReferenceAnswer string

ReferenceAnswer is an author-supplied golden answer used by evaluators that compare against a reference. A valid ReferenceAnswer is non-empty, valid UTF-8, and within MaxReferenceAnswerBytes.

func (r ReferenceAnswer) Validate() error

type Report ¶

type Report struct {
	ID         string
	Suite      Revision
	Target     Revision
	StartedAt  time.Time
	EndedAt    time.Time
	Samples    []SampleReport
	Summary    Summary
	Provenance Provenance
}

Report is the complete result of one Run: the samples in deterministic order, a minimal status summary, and the provenance needed to interpret and compare it. ID is a deterministic identifier derived from the suite (see Suite.reportID); a caller may overwrite it with a globally unique run ID. Suite is the suite revision; Target is the observed target revision, kept distinct from Suite so a report records both what was run and what it ran against.

func Run(ctx context.Context, cfg RunConfig, suite Suite, target Target, evaluators ...Evaluator) (Report, error)

func (r Report) Validate() error

type ReportValidationError ¶

type ReportValidationError struct {
	// Reason is one of the reportReason* constants.
	Reason string
}

ReportValidationError reports that a Report failed a report-level invariant (identity, timestamp ordering, trial index, sample or evaluator uniqueness, summary consistency, or observed-target consistency). Reason is drawn only from the fixed vocabulary below, so no untrusted content — in particular a data-supplied scenario ID — is ever embedded; only the class of failure is reported.

func (e *ReportValidationError) Error() string

type Revision ¶

type Revision string

Revision identifies a specific version of a named object, such as an evaluator revision, a model revision, or a prompt revision. A valid Revision is non-empty, valid UTF-8, and no longer than MaxRevisionBytes bytes.

func (r Revision) Validate() error

type RunConfig ¶

type RunConfig struct {
	// Trials is how many times each scenario is executed. A value of 0 means one
	// trial. Negative or above MaxTrials is rejected at preflight.
	Trials int
	// Concurrency is the maximum number of samples executed simultaneously. A
	// value of 0 (or 1) means sequential execution. Negative is rejected.
	// Concurrency is opt-in: the default never runs targets in parallel.
	Concurrency int
	// TargetTimeout bounds a single target execution. Zero means no per-target
	// timeout (the run context still applies). Negative is rejected.
	TargetTimeout time.Duration
	// EvaluatorTimeout bounds a single evaluator execution. Zero means no
	// per-evaluator timeout. Negative is rejected.
	EvaluatorTimeout time.Duration
	// contains filtered or unexported fields
}

RunConfig shapes one execution: how many trials per scenario, how much target concurrency to allow, and the per-stage timeouts. Its zero value is a valid, conservative default — one trial, sequential, no per-stage timeout — so RunConfig{} runs a suite deterministically.

func (c RunConfig) Validate() error

type Sample ¶

type Sample struct {
	Scenario    *Scenario
	Observation Observation
}

Sample binds a scenario to the observation it produced so an evaluator can assess them together. Scenario is nil for pure continuous observation, where an observation already exists and no target was executed. Observation is always present.

func (s Sample) Validate() error

type SampleReport ¶

type SampleReport struct {
	ScenarioID  string
	TrialIndex  int
	Observation Observation
	TargetErr   *TargetError
	Assessments []Assessment
}

SampleReport is the result of one (scenario, trial) sample. ScenarioID and TrialIndex are the sample's stable derived identity. Observation is what the target produced (the zero Observation when the target failed). TargetErr is a typed stage error when the target stage failed, and nil when it succeeded — a target failure is recorded here, never as a failed assessment. Assessments holds each evaluator's individual result in evaluator order; a per-evaluator failure appears as an error-status assessment beside its succeeding siblings, never discarding them.

type SampleSubjectMismatchError ¶

type SampleSubjectMismatchError struct{}

SampleSubjectMismatchError reports that a sample's observation described a subject whose revision did not match the target revision the sample's scenario declares. This is a stage error — the target produced an observation for the wrong revision — not a failed assessment. Both revisions are withheld from the message: the subject revision originates with the target and must not leak through a diagnostic.

func (e *SampleSubjectMismatchError) Error() string

type Scenario ¶

type Scenario struct {
	ID          string
	Name        Name
	Revision    Revision
	Input       content.AgenticMessages
	Expectation *Expectation
	Labels      []Label
}

Scenario is an active qualification case: a stable identity, the input thread to drive a target with, optional qualification expectations, and labels. ID is the stable case identity used to reject duplicate scenarios and correlate reports across runs; Name and Revision identify the target revision the scenario qualifies (see Sample.Validate). Input must be non-empty — an empty scenario drives nothing.

func (s Scenario) Validate() error

type Scope ¶

type Scope uint8

Scope names the observational granularity an assessment applies to.

const (
	// ScopeCase covers a single qualification case. It is the valid zero value.
	ScopeCase Scope = iota
	// ScopeTurn covers one completed turn.
	ScopeTurn
	// ScopeSession covers a full session.
	ScopeSession
	// ScopeRun covers an entire run of many sessions or cases.
	ScopeRun
)

func (s Scope) Validate() error

type Severity ¶

type Severity string

Severity ranks a Finding. The members are ordered from least to most serious as declared here (info < low < medium < high < critical). It is a string type for direct rendering. There is no valid zero value: an unset severity does not validate.

const (
	// SeverityInfo is informational, not a defect.
	SeverityInfo Severity = "info"
	// SeverityLow is a minor issue.
	SeverityLow Severity = "low"
	// SeverityMedium is a moderate issue.
	SeverityMedium Severity = "medium"
	// SeverityHigh is a serious issue.
	SeverityHigh Severity = "high"
	// SeverityCritical is the most serious issue.
	SeverityCritical Severity = "critical"
)

func (s Severity) Validate() error

type Sink ¶

type Sink interface {
	WriteReport(context.Context, Report) error
}

Sink is the destination contract for a completed Report. It is the core reporting seam: the runner (or a continuous-eval loop) hands each Report to a Sink, and a Sink persists, forwards, or exports it. Implementations live outside the root package (a JSON file sink, an OpenTelemetry sink, an in-memory sink) so the root never depends on a storage or wire technology. WriteReport takes a context so a slow or networked sink is cancellable, and returns a typed error the caller can classify. A Sink must treat the Report as untrusted-adjacent: it must not place raw conversation text, judge explanations, or secrets in any external label — redaction is the wire form's responsibility, enforced by the concrete sink.

type StatusConsistencyError ¶

type StatusConsistencyError struct {
	// Status is the assessment's declared status.
	Status AssessmentStatus
	// Reason is one of the statusReason* constants.
	Reason string
}

StatusConsistencyError reports that an Assessment's declared Status was inconsistent with its contents — for example a pass carrying a high/critical-severity finding, or a non-verdict status (unverified, error, skipped) carrying a quality measurement. Status is the assessment's own validated status (a closed enum constant) and Reason is drawn only from the fixed vocabulary below, so no untrusted content is embedded.

func (e *StatusConsistencyError) Error() string

type StructuredErrorReason ¶

type StructuredErrorReason string

StructuredErrorReason classifies why a structured-output response failed validation. There is no valid zero value: an unclassified failure is rejected so a bare error can never masquerade as a known, benign one.

const (
	// StructuredErrorInvalidJSON: the response was not valid JSON.
	StructuredErrorInvalidJSON StructuredErrorReason = "invalid_json"
	// StructuredErrorSchemaMismatch: the response did not match the schema shape.
	StructuredErrorSchemaMismatch StructuredErrorReason = "schema_mismatch"
	// StructuredErrorMissingField: a required field was absent.
	StructuredErrorMissingField StructuredErrorReason = "missing_field"
	// StructuredErrorOutOfRange: a value fell outside its permitted range.
	StructuredErrorOutOfRange StructuredErrorReason = "out_of_range"
	// StructuredErrorEmptyOutput: the model produced no terminal output.
	StructuredErrorEmptyOutput StructuredErrorReason = "empty_output"
)

func (r StructuredErrorReason) Validate() error

type StructuredOutput ¶

type StructuredOutput struct {
	SchemaName     Name
	SchemaRevision Revision
}

StructuredOutput is the positive evidence that the subject produced structured output that validated against its declared schema. It carries only safe schema identity — the schema's name and/or revision — and never the raw model output, so it is a content-free proof of structured-output success. Both identity fields are optional (the presence of this evidence is itself the signal), and each is validated only when supplied.

type StructuredOutputError ¶

type StructuredOutputError struct {
	Schema     Revision
	Reason     StructuredErrorReason
	DetailHash ContentHash
}

StructuredOutputError records a structured-output validation failure as a classification plus an optional correlatable hash of the offending detail. The raw model text is never stored.

type StructuredOutputExpectation ¶

type StructuredOutputExpectation struct {
	Schema Revision
	Strict bool
}

StructuredOutputExpectation asserts that the interaction produces a terminal structured output conforming to a named schema revision. Schema is required (an expectation with no schema asserts nothing and is malformed). Strict mirrors the inference OutputSchema.Strict flag: when true the output must match the schema exactly.

func (s StructuredOutputExpectation) Validate() error

type Subject ¶

type Subject struct {
	ID       string
	Kind     SubjectKind
	Name     Name
	Revision Revision
}

Subject identifies what produced the observation. ID is a required correlation identifier; Name and Revision reuse the domain identity types so the same validation rules apply everywhere.

func (s Subject) Validate() error

type SubjectKind ¶

type SubjectKind string

SubjectKind names what is under evaluation. There is no valid zero value: a subject must declare what it is, so an unset kind is rejected and cannot be silently treated as, say, a model. The member set covers the Phase-1 targets the design names and nothing more.

const (
	// SubjectModel is a bare inference model.
	SubjectModel SubjectKind = "model"
	// SubjectAgent is an agent entry point.
	SubjectAgent SubjectKind = "agent"
	// SubjectPrompt is a prompt or prompt template.
	SubjectPrompt SubjectKind = "prompt"
	// SubjectHTTPEndpoint is an HTTP service under test.
	SubjectHTTPEndpoint SubjectKind = "http_endpoint"
	// SubjectProcess is a local process under test.
	SubjectProcess SubjectKind = "process"
)

func (k SubjectKind) Validate() error

type Suite ¶

type Suite struct {
	Name      Name
	Revision  Revision
	Scenarios []Scenario
}

Suite is a named, versioned set of scenarios to run against a single target. Name and Revision identify the suite for provenance and reporting (Revision becomes Report.Suite); Scenarios is the ordered set of cases. Scenario order is preserved in the report, so a suite is reproducible.

func (s Suite) Validate() error

type Summary ¶

type Summary struct {
	Samples      int
	TargetErrors int
	Assessments  map[AssessmentStatus]int
}

Summary is a minimal roll-up of a report: the sample count, how many samples failed at the target stage, and a count of assessments by status. It is kept deliberately small; distribution, quantile, and baseline-comparison reporting are added by a later task and do not belong in the runner.

type Target ¶

type Target interface {
	Name() string
	Observe(context.Context, Scenario) (Observation, error)
}

Target executes a scenario and reports the resulting observation. Name identifies the target for provenance and reporting. Observe runs the scenario and returns its observation; it takes a context so callers can bound execution, and it returns a typed error on failure rather than encoding failure as an observation. A target error is a stage error and must never be reported as a failed assessment.

Observe's Scenario argument is READ-ONLY. The passed Scenario and everything it references — its Input, Labels, and Expectation — share backing with the caller's suite; an implementation MUST NOT mutate them. The runner shallow- copies the Scenario struct header only and relies on this contract in place of a deep copy. A target that needs to derive a modified scenario must copy what it changes.

type TargetError ¶

type TargetError struct {
	// Cause is the underlying failure. It may be a context error, a target's own
	// error, or an observation ValidationError.
	Cause error
}

TargetError reports that a sample's target stage failed: the target returned an error, timed out, was cancelled, or produced an observation that did not validate. It is a stage error, never a failed quality assessment. The wrapped Cause is available via Unwrap so callers can classify the failure (for example errors.Is(err, context.DeadlineExceeded)); the Error() text is fixed and never echoes the cause, which may originate outside the process and carry untrusted content.

func (e *TargetError) Error() string

func (e *TargetError) Unwrap() error

type TimingEvidence ¶

type TimingEvidence struct {
	Label    Name
	Duration time.Duration
}

TimingEvidence records the duration of a step. Duration is a safe scalar.

type ToolCallExpectation ¶

type ToolCallExpectation struct {
	Tool     Name
	MinCount int
	MaxCount *int
}

ToolCallExpectation asserts that a named tool is invoked a bounded number of times. Tool is required. MinCount is the inclusive lower bound and must not be negative. MaxCount is an optional inclusive upper bound; when set it must be non-negative and no less than MinCount. A nil MaxCount leaves the count unbounded above.

func (t ToolCallExpectation) Validate() error

type ToolOperationEvidence ¶

type ToolOperationEvidence struct {
	ToolName    Name
	ToolUseID   string
	ArgsHash    ContentHash
	ArgsBytes   int
	ResultBytes int
	IsError     bool
}

ToolOperationEvidence records a tool invocation as safe metadata only. Tool arguments and results are sensitive, so they are represented by a hash and byte counts — never stored raw on this value.

type Trace ¶

type Trace struct {
	TraceID       string
	SessionID     string
	TurnID        string
	StartedAt     time.Time
	EndedAt       time.Time
	Model         Revision
	Prompt        Revision
	MessageRanges []MessageRange
	Operations    []Operation
	Evidence      []Evidence
}

Trace holds facts not already present in the conversation: correlation identifiers, timing, the model and prompt revisions in effect, message ranges, operations, and typed evidence.

type Unit ¶

type Unit string

Unit names the dimension of a Measurement.Value. The member set covers the dimensions the Phase-1 evaluators produce and nothing more. There is no valid zero value: every measurement must declare a known unit.

const (
	// UnitCount is a dimensionless whole-number tally.
	UnitCount Unit = "count"
	// UnitRatio is a dimensionless proportion or rate.
	UnitRatio Unit = "ratio"
	// UnitSecond is a duration in seconds.
	UnitSecond Unit = "second"
	// UnitToken is a count of model tokens.
	UnitToken Unit = "token"
	// UnitByte is a size in bytes.
	UnitByte Unit = "byte"
)

func (u Unit) Validate() error

type UnknownEvidenceError ¶

type UnknownEvidenceError struct{}

UnknownEvidenceError reports that an EvidenceRef pointed at an EvidenceID that no evidence entry in the trace defines. The dangling identifier is withheld for the same reason as DuplicateEvidenceError.

func (e *UnknownEvidenceError) Error() string

type UsageEvidence ¶

type UsageEvidence struct {
	Model Revision
	Usage content.Usage
}

UsageEvidence records model token usage. All fields are safe counts.

type ValidationError ¶

type ValidationError struct {
	// Field is the domain type or field name, e.g. "Name" or "Revision".
	Field string
	// Reason is a bounded, safe explanation, e.g. "must not be empty".
	Reason string
}

ValidationError reports that a domain value failed validation. Field names the domain type or field that failed; Reason is a short, developer-facing explanation drawn only from a fixed vocabulary of package constants and bounds. Neither field ever contains the offending value, so an untrusted or oversized input cannot leak through an error.

func (e *ValidationError) Error() string