handlers

type A2AEvalHandler ¶

type A2AEvalHandler struct{}

A2AEvalHandler evaluates a single assistant turn by sending conversation context to an A2A agent and interpreting the agent's response as a structured eval result.

Params:

• agent_url (string, required): A2A agent endpoint URL
• auth_token (string, optional): auth token, supports ${ENV_VAR}
• timeout (string, optional): request timeout, default 60s
• criteria (string, optional): evaluation criteria
• include_messages (bool, optional): include conversation history, default true
• include_tool_calls (bool, optional): include tool call records, default false
• min_score (float64, optional): minimum score threshold
• extra (map[string]any, optional): arbitrary data forwarded in request

func (h *A2AEvalHandler) Eval(
	ctx context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *A2AEvalHandler) Type() string

type A2AEvalSessionHandler ¶

type A2AEvalSessionHandler struct{}

A2AEvalSessionHandler evaluates an entire conversation by sending all assistant messages to an A2A agent.

Params: same as A2AEvalHandler.

func (h *A2AEvalSessionHandler) Eval(
	ctx context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *A2AEvalSessionHandler) Type() string

type AgentInvokedHandler ¶

type AgentInvokedHandler struct{}

AgentInvokedHandler checks that expected agents were invoked as tool calls. Params: agents []string — list of agent names that should have been called.

func (h *AgentInvokedHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *AgentInvokedHandler) Type() string

type AgentNotInvokedHandler ¶

type AgentNotInvokedHandler struct{}

AgentNotInvokedHandler checks that forbidden agents were NOT called. Params: agents []string — agent names that should not have been called.

func (h *AgentNotInvokedHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *AgentNotInvokedHandler) Type() string

type AgentResponseContainsHandler ¶

type AgentResponseContainsHandler struct{}

AgentResponseContainsHandler checks that a specific agent's response contains expected text. Agent responses appear as tool-result messages where the tool name matches the agent name. Params: agent string, contains string.

func (h *AgentResponseContainsHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *AgentResponseContainsHandler) Type() string

type AnswerRelevancyHandler ¶

type AnswerRelevancyHandler struct{}

AnswerRelevancyHandler scores how directly the assistant's answer addresses the user's question. Equivalent in name to DeepEval / Ragas `answer_relevancy`.

Default prompts adapted from the public DeepEval / Ragas reference implementations (Apache 2.0). Override per-call by passing system_prompt or criteria in params.

Params (all optional):

• question (string): override the auto-extracted last user turn
• rubric, model, system_prompt: standard llm_judge knobs

func (h *AnswerRelevancyHandler) Eval(
	ctx context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *AnswerRelevancyHandler) Type() string

type AudioDurationHandler ¶

type AudioDurationHandler struct{}

AudioDurationHandler checks that audio duration is within range. Params: min_seconds float64, max_seconds float64.

func (h *AudioDurationHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *AudioDurationHandler) Type() string

type AudioEmotionHandler ¶

type AudioEmotionHandler struct{}

AudioEmotionHandler is a pure eval primitive: it calls the AudioClassifier resolved from the orchestrator's classify registry, picks the score for the chosen expected_label, and emits it as EvalResult.Score. Threshold judgment (min_score / max_score) lives on `type: assertion` wrappers — NOT on this handler.

Wrap with `type: assertion` to assert against a threshold:

• type: assertion params: eval_type: audio_emotion eval_params: { model: "...", expected_label: "ang", message_role: user } min_score: 0.5

Use directly in pack `evals:` to emit the raw signal at runtime (for metrics / observability).

Params:

• model string (required) — backend model id, e.g. "superb/wav2vec2-base-superb-er"
• expected_label string (required) — label whose score is emitted
• message_role string (optional, default "user") — which speaker's audio to score
• message_index int (optional, default -1 = latest match) — pick a specific audio message
• classifier_id string (optional) — explicit registry id; empty uses the configured default

Putting min_score / max_score on this handler is rejected — the assertion wrapper is the canonical home for thresholds.

func (h *AudioEmotionHandler) Eval(
	ctx context.Context, evalCtx *evals.EvalContext, params map[string]any,
) (*evals.EvalResult, error)

func (h *AudioEmotionHandler) Type() string

type AudioFormatHandler ¶

type AudioFormatHandler struct{}

AudioFormatHandler checks that audio content has allowed formats. Params: formats []string.

func (h *AudioFormatHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *AudioFormatHandler) Type() string

type BiasHandler ¶

type BiasHandler struct{}

BiasHandler scores assistant output for demographic / stereotype bias via the LLM judge. DeepEval-equivalent name; default-wired as a guardrail. Params: standard llm_judge knobs.

func (h *BiasHandler) Eval(
	ctx context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *BiasHandler) Type() string

type CompositionBranchTakenHandler ¶

type CompositionBranchTakenHandler struct{}

CompositionBranchTakenHandler asserts a branch step took the expected target. Params: branch (string, required), expected (string, required).

func (h *CompositionBranchTakenHandler) Eval(
	_ context.Context, evalCtx *evals.EvalContext, params map[string]any,
) (*evals.EvalResult, error)

func (h *CompositionBranchTakenHandler) Type() string

type CompositionOutputHandler ¶

type CompositionOutputHandler struct{}

CompositionOutputHandler asserts the composition's final output contains/equals a value. Params: one of contains|equals (string).

func (h *CompositionOutputHandler) Eval(
	_ context.Context, evalCtx *evals.EvalContext, params map[string]any,
) (*evals.EvalResult, error)

func (h *CompositionOutputHandler) Type() string

type CompositionParallelCompleteHandler ¶

type CompositionParallelCompleteHandler struct{}

CompositionParallelCompleteHandler asserts a parallel step completed. Params: parallel (string, required).

func (h *CompositionParallelCompleteHandler) Eval(
	_ context.Context, evalCtx *evals.EvalContext, params map[string]any,
) (*evals.EvalResult, error)

func (h *CompositionParallelCompleteHandler) Type() string

type CompositionStepOutputHandler ¶

type CompositionStepOutputHandler struct{}

CompositionStepOutputHandler asserts a named step's output contains/equals a value. Params: step (string, required); one of contains|equals (string).

func (h *CompositionStepOutputHandler) Eval(
	_ context.Context, evalCtx *evals.EvalContext, params map[string]any,
) (*evals.EvalResult, error)

func (h *CompositionStepOutputHandler) Type() string

type ContainsAnyHandler ¶

type ContainsAnyHandler struct{}

ContainsAnyHandler checks that at least one assistant message contains at least one of the specified patterns. Params: patterns []string (case-insensitive matching).

func (h *ContainsAnyHandler) Eval(
	ctx context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (_ *evals.EvalResult, _ error)

func (h *ContainsAnyHandler) Type() string

type ContainsHandler ¶

type ContainsHandler struct{}

ContainsHandler checks if CurrentOutput contains all specified patterns (case-insensitive). Params: patterns []string.

func (h *ContainsHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (result *evals.EvalResult, err error)

func (h *ContainsHandler) Type() string

type ContentExcludesHandler ¶

type ContentExcludesHandler struct{}

ContentExcludesHandler checks that NONE of the assistant messages across the full conversation contain any of the forbidden patterns. Params: patterns []string (case-insensitive matching).

func (h *ContentExcludesHandler) Eval(
	ctx context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (_ *evals.EvalResult, _ error)

func (h *ContentExcludesHandler) EvalPartial(
	_ context.Context, content string, params map[string]any,
) (*evals.EvalResult, error)

func (h *ContentExcludesHandler) Type() string

type ContextualPrecisionHandler ¶

type ContextualPrecisionHandler struct{}

ContextualPrecisionHandler scores the fraction of retrieved chunks that are actually relevant to the question. Equivalent in name to DeepEval `contextual_precision`.

Default prompts adapted from the public DeepEval reference implementation (Apache 2.0). Override per-call by passing system_prompt or criteria in params.

Params (all optional):

• contexts ([]string) | context (string) | context_field (string): retrieved chunks
• question (string): override the auto-extracted last user turn
• rubric, model, system_prompt: standard llm_judge knobs

func (h *ContextualPrecisionHandler) Eval(
	ctx context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *ContextualPrecisionHandler) Type() string

type ContextualRecallHandler ¶

type ContextualRecallHandler struct{}

ContextualRecallHandler scores how completely the retrieved chunks cover the information needed for the ground-truth answer. Equivalent in name to DeepEval / Ragas `contextual_recall`.

Default prompts adapted from the public DeepEval / Ragas reference implementations (Apache 2.0). Override per-call by passing system_prompt or criteria in params.

Params:

• contexts ([]string) | context (string) | context_field (string): retrieved chunks (required)
• reference (string) | expected_output (string): the ground-truth answer (required)
• question (string): override the auto-extracted last user turn
• rubric, model, system_prompt: standard llm_judge knobs

func (h *ContextualRecallHandler) Eval(
	ctx context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *ContextualRecallHandler) Type() string

type ContextualRelevancyHandler ¶

type ContextualRelevancyHandler struct{}

ContextualRelevancyHandler scores the average per-chunk relevance of the retrieved chunks to the question. Equivalent in name to DeepEval `contextual_relevancy`.

Distinct from `contextual_precision`: precision is a binary relevant/not-relevant ratio; relevancy is the mean of graded scores.

Default prompts adapted from the public DeepEval reference implementation (Apache 2.0). Override per-call by passing system_prompt or criteria in params.

Params (all optional):

• contexts ([]string) | context (string) | context_field (string)
• question (string)
• rubric, model, system_prompt

func (h *ContextualRelevancyHandler) Eval(
	ctx context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *ContextualRelevancyHandler) Type() string

type CosineSimilarityHandler ¶

type CosineSimilarityHandler struct{}

CosineSimilarityHandler computes cosine similarity between embeddings. Params: reference []float64, min_similarity float64. Target embedding comes from Metadata["embedding"].

func (h *CosineSimilarityHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (result *evals.EvalResult, err error)

func (h *CosineSimilarityHandler) Type() string

type CostBudgetHandler ¶

type CostBudgetHandler struct{}

CostBudgetHandler checks conversation-level cost and token limits. Params:

• max_cost_usd: float64 — maximum total cost in USD (optional)
• max_input_tokens: int — maximum input tokens (optional)
• max_output_tokens: int — maximum output tokens (optional)
• max_total_tokens: int — maximum total tokens (optional)

func (h *CostBudgetHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *CostBudgetHandler) Type() string

type DirectionalHandler ¶

type DirectionalHandler struct{}

DirectionalHandler compares a run's output against a baseline expectation. Used in behavioral testing (Phase 6) to verify perturbation-invariant behavior. Params:

• check (string, required): "same_tool_calls", "same_outcome", or "similar_content"
• baseline_tools ([]string, optional): expected tool names for same_tool_calls
• baseline_state (string, optional): expected workflow state for same_outcome
• baseline_content (string, optional): expected content substring for similar_content
• threshold (float64, optional): minimum overlap ratio for similar_content (default 0.5)

func (h *DirectionalHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *DirectionalHandler) Type() string

type ExecEvalConfig ¶

type ExecEvalConfig struct {
	TypeName  string
	Command   string
	Args      []string
	Env       []string
	TimeoutMs int
}

ExecEvalConfig holds the configuration for creating an ExecEvalHandler.

type ExecEvalHandler ¶

type ExecEvalHandler struct {
	// contains filtered or unexported fields
}

ExecEvalHandler evaluates content by spawning an external subprocess. The subprocess receives an ExecEvalRequest as JSON on stdin and must return an ExecEvalResponse as JSON on stdout.

ExecEvalHandler is registered dynamically from RuntimeConfig exec bindings, not via init(). Each binding creates a handler whose Type() matches the eval type name in the pack — making exec evals transparent to the pack.

func NewExecEvalHandler(cfg *ExecEvalConfig) *ExecEvalHandler

func (h *ExecEvalHandler) Eval(
	ctx context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *ExecEvalHandler) Type() string

type ExternalEvalRequest ¶

type ExternalEvalRequest struct {
	CurrentOutput string         `json:"current_output"`
	Messages      []messageView  `json:"messages,omitempty"`
	ToolCalls     []toolCallView `json:"tool_calls,omitempty"`
	Criteria      string         `json:"criteria,omitempty"`
	Variables     map[string]any `json:"variables,omitempty"`
	Extra         map[string]any `json:"extra,omitempty"`
}

ExternalEvalRequest is the standard request body sent to external eval endpoints (REST) and formatted as context for A2A eval agents.

type FaithfulnessHandler ¶

type FaithfulnessHandler struct{}

FaithfulnessHandler scores how well the answer is supported by the provided context. A faithful answer makes only claims backed by the context; an unfaithful answer asserts facts the context does not support. Equivalent in name to DeepEval / Ragas `faithfulness`.

Default prompts adapted from the public DeepEval / Ragas reference implementations (Apache 2.0). Override per-call by passing system_prompt or criteria in params.

Params (all optional):

• contexts ([]string) | context (string) | context_field (string): retrieved chunks the answer should be grounded in
• rubric, model, system_prompt: standard llm_judge knobs

func (h *FaithfulnessHandler) Eval(
	ctx context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *FaithfulnessHandler) Type() string

type FieldPresenceHandler ¶

type FieldPresenceHandler struct{}

FieldPresenceHandler checks that required fields are present in the output. Params: fields []string (field names to look for, case-insensitive).

func (h *FieldPresenceHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *FieldPresenceHandler) Type() string

type GuardrailTriggeredHandler ¶

type GuardrailTriggeredHandler struct{}

GuardrailTriggeredHandler checks if a specific eval or guardrail triggered (or didn't trigger) as expected.

It searches EvalContext.PriorResults for an eval whose Type or EvalID matches the validator_type parameter. Pipeline-level guardrail results from message.Validations are automatically seeded into PriorResults by BuildEvalContext, so all guardrail outcomes are available through a single lookup path.

Params: validator_type string, should_trigger bool (default true), direction string (optional).

direction narrows the search to firings the guardrail recorder stamped with that side: "input" (judged the user's message, before the call) or "output" (judged the assistant response). Omitted — or "both" — matches a firing in either direction, which is the historical behavior and stays byte-identical. Set it when an input and an output guardrail of the same eval type are both declared: without it only the last recorded firing is reachable, so neither can be targeted (#1718).

func (h *GuardrailTriggeredHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *GuardrailTriggeredHandler) Type() string

func (h *GuardrailTriggeredHandler) ValidateParams(params map[string]any) error

type HallucinationHandler ¶

type HallucinationHandler struct{}

HallucinationHandler scores how free the answer is of claims that are not supported by, or contradict, the provided context. The inverse framing of `faithfulness` — kept as a separate handler so users coming from DeepEval find the vocabulary they expect.

A score of 1.0 means no hallucination (matches faithfulness=1.0); a score of 0.0 means entirely hallucinated.

Default prompts adapted from the public DeepEval reference implementation (Apache 2.0). Override per-call by passing system_prompt or criteria in params.

Params (all optional):

• contexts ([]string) | context (string) | context_field (string)
• rubric, model, system_prompt: standard llm_judge knobs

func (h *HallucinationHandler) Eval(
	ctx context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *HallucinationHandler) Type() string

type ImageDimensionsHandler ¶

type ImageDimensionsHandler struct{}

ImageDimensionsHandler checks that images meet dimension requirements. Params: min_width, max_width, min_height, max_height, width, height.

func (h *ImageDimensionsHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *ImageDimensionsHandler) Type() string

type ImageFormatHandler ¶

type ImageFormatHandler struct{}

ImageFormatHandler checks that images in assistant messages have allowed formats. Params: formats []string.

func (h *ImageFormatHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *ImageFormatHandler) Type() string

type ImageModerationHandler ¶

type ImageModerationHandler struct{}

ImageModerationHandler is a pure eval primitive: it runs the latest image in the target role's turns through the configured ImageClassifier and emits the score for expected_label (e.g. "nsfw") as EvalResult.Score. Threshold judgment (min_score / max_score) lives on `type: assertion` / `type: guardrail` wrappers — NOT on this handler.

Params:

• model string (required) — classifier model id, e.g. "Falconsai/nsfw_image_detection"
• expected_label string (required) — label whose score is emitted
• message_role string (optional, default "assistant") — whose image to score
• message_index int (optional, default -1 = latest match)
• classifier_id string (optional) — explicit registry id; empty uses the configured default

func (h *ImageModerationHandler) Eval(
	ctx context.Context, evalCtx *evals.EvalContext, params map[string]any,
) (*evals.EvalResult, error)

func (h *ImageModerationHandler) Type() string

type InvariantFieldsPreservedHandler ¶

type InvariantFieldsPreservedHandler struct{}

InvariantFieldsPreservedHandler checks that field values in tool call arguments are not lost between calls to the same tool. If a field was present in an earlier call but disappears in a later call, that is a violation.

Params:

• tool (string, required): The tool name to track across calls.
• fields ([]string, required): JSON field names to track in tool call arguments.

func (h *InvariantFieldsPreservedHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *InvariantFieldsPreservedHandler) Type() string

type JSONPathHandler ¶

type JSONPathHandler struct{}

JSONPathHandler validates assistant output as JSON using JMESPath expressions. Params:

• expression string (JMESPath expression)
• expected any (optional: exact match)
• contains []any (optional: array contains check)
• min_results int, max_results int (optional: array length bounds)
• min float64, max float64 (optional: numeric range)
• allow_wrapped bool (optional: extract JSON from code blocks)
• extract_json bool (optional: extract JSON from mixed text)

func (h *JSONPathHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *JSONPathHandler) Type() string

type JSONSchemaHandler ¶

type JSONSchemaHandler struct{}

JSONSchemaHandler validates CurrentOutput against a JSON schema. Params: schema map[string]any.

func (h *JSONSchemaHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (result *evals.EvalResult, err error)

func (h *JSONSchemaHandler) Type() string

type JSONValidHandler ¶

type JSONValidHandler struct{}

JSONValidHandler checks if CurrentOutput is valid JSON. Params: allow_wrapped bool, extract_json bool (both optional).

func (h *JSONValidHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (result *evals.EvalResult, err error)

func (h *JSONValidHandler) Type() string

type JudgeOpts ¶

type JudgeOpts struct {
	// Content is the text being evaluated (assistant response or full conversation).
	Content string

	// Criteria describes what the judge should evaluate (e.g. "Is the response helpful?").
	Criteria string

	// Rubric provides detailed scoring guidance (optional).
	Rubric string

	// Model specifies which model to use for judging (optional, provider decides default).
	Model string

	// SystemPrompt overrides the default judge system prompt (optional).
	SystemPrompt string

	// Extra holds additional parameters for provider-specific features.
	Extra map[string]any

	// Emitter is an optional event emitter for provider call telemetry.
	Emitter *events.Emitter
}

JudgeOpts configures a judge evaluation request.

type JudgeProvider ¶

type JudgeProvider interface {
	// Judge sends the evaluation prompt to an LLM and returns
	// the parsed verdict. Implementations handle provider selection,
	// prompt formatting, and response parsing.
	Judge(ctx context.Context, opts JudgeOpts) (*JudgeResult, error)
}

JudgeProvider abstracts LLM access for judge-based evaluations. Arena, SDK, and eval workers each provide their own implementation wiring their respective provider infrastructure.

type JudgeResult ¶

type JudgeResult struct {
	// Passed indicates whether the content met the evaluation criteria.
	Passed bool

	// Score is the numerical score assigned by the judge (typically 0.0-1.0).
	Score float64

	// Reasoning explains the judge's evaluation.
	Reasoning string

	// Raw is the unprocessed LLM response text.
	Raw string
}

JudgeResult captures the output of an LLM judge evaluation.

type LLMJudgeHandler ¶

type LLMJudgeHandler struct{}

LLMJudgeHandler evaluates a single assistant turn using an LLM judge. Pure eval primitive (see docs/reference/checks.md for the `type: assertion` wrapper pattern that adds thresholds). The JudgeProvider must be supplied in evalCtx.Metadata["judge_provider"].

Params:

• criteria (string, required): what to evaluate
• rubric (string, optional): detailed scoring guidance
• model (string, optional): model override for the judge
• system_prompt (string, optional): override default system prompt

func (h *LLMJudgeHandler) Eval(
	ctx context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (result *evals.EvalResult, err error)

func (h *LLMJudgeHandler) Type() string

type LLMJudgeSessionHandler ¶

type LLMJudgeSessionHandler struct{}

LLMJudgeSessionHandler is the session-level counterpart of LLMJudgeHandler. It runs the judge once over a full, role-labeled transcript of the conversation — user (and other) turns, assistant text, and every tool call with its arguments and result — so the judge sees what the agent actually did, not only its prose. Same params, same conventions. Registered under `llm_judge_conversation` too (see register.go).

func (h *LLMJudgeSessionHandler) Eval(
	ctx context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (result *evals.EvalResult, err error)

func (h *LLMJudgeSessionHandler) Type() string

type LLMJudgeToolCallsHandler ¶

type LLMJudgeToolCallsHandler struct{}

LLMJudgeToolCallsHandler is the tool-call counterpart of LLMJudgeHandler — feeds tool call data (names, args, results) instead of the assistant's text. Accepts the same base params plus `tools []string` to filter to specific tool names.

func (h *LLMJudgeToolCallsHandler) Eval(
	ctx context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *LLMJudgeToolCallsHandler) Type() string

type LatencyBudgetHandler ¶

type LatencyBudgetHandler struct{}

LatencyBudgetHandler checks Metadata["latency_ms"] against a max. Params: max_ms float64.

func (h *LatencyBudgetHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (result *evals.EvalResult, err error)

func (h *LatencyBudgetHandler) Type() string

type MaxLengthHandler ¶

type MaxLengthHandler struct{}

MaxLengthHandler checks that CurrentOutput does not exceed the specified character count. Accepts params: max or max_characters (int).

func (h *MaxLengthHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *MaxLengthHandler) EvalPartial(
	_ context.Context, content string, params map[string]any,
) (*evals.EvalResult, error)

func (h *MaxLengthHandler) Type() string

func (h *MaxLengthHandler) ValidateParams(params map[string]any) error

type MinLengthHandler ¶

type MinLengthHandler struct{}

MinLengthHandler checks that CurrentOutput has at least the specified character count. Accepts params: min or min_characters (int).

func (h *MinLengthHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *MinLengthHandler) Type() string

func (h *MinLengthHandler) ValidateParams(params map[string]any) error

type NoToolErrorsHandler ¶

type NoToolErrorsHandler struct{}

NoToolErrorsHandler checks that no tool calls returned errors. Params: tools []string (optional) — if set, only checks calls matching those tool names.

func (h *NoToolErrorsHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *NoToolErrorsHandler) Type() string

type OutcomeEquivalentHandler ¶

type OutcomeEquivalentHandler struct{}

OutcomeEquivalentHandler checks that a single run's outcome matches an expected value. Used in behavioral testing (Phase 6) to verify perturbation-invariant outcomes. Params:

• metric (string, required): "tool_calls", "final_state", or "content_hash"
• expected_tools ([]string, optional): for tool_calls metric
• expected_state (string, optional): for final_state metric
• expected_content (string, optional): for content_hash metric

func (h *OutcomeEquivalentHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *OutcomeEquivalentHandler) Type() string

type PIILeakageHandler ¶

type PIILeakageHandler struct{}

PIILeakageHandler scores whether the assistant output leaks personally-identifiable information. Equivalent in name to DeepEval `pii_leakage`. Default wiring in this codebase is as a guardrail (pack `validators:` block); scenarios observe firing via `guardrail_triggered`. The runtime guardrail enforces (blocks / replaces) the offending content; the assertion observes the firing.

Implementation runs a regex pre-pass for high-confidence patterns (emails, US SSN, 16-digit card-shape numbers) before the LLM-judged path. On regex match, the handler returns score 0 immediately without an LLM call — keeps the obvious cases cheap and deterministic. On miss, the LLM judge inspects the answer for ambiguous PII (names tied to other PII, less-strict patterns, etc.).

Default prompts adapted from the public DeepEval reference implementation (Apache 2.0).

Params (all optional):

• rubric, model, system_prompt, criteria: standard llm_judge knobs

func (h *PIILeakageHandler) Eval(
	ctx context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *PIILeakageHandler) Type() string

type RegexHandler ¶

type RegexHandler struct{}

RegexHandler checks if CurrentOutput matches a regex pattern. Params: pattern string.

func (h *RegexHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (result *evals.EvalResult, err error)

func (h *RegexHandler) Type() string

type RestEvalHandler ¶

type RestEvalHandler struct{}

RestEvalHandler evaluates a single assistant turn by POSTing conversation context to an external HTTP endpoint and interpreting the structured JSON response.

Params:

• url (string, required): endpoint URL
• method (string, optional): HTTP method, default POST
• headers (map[string]string, optional): request headers, supports ${ENV_VAR}
• timeout (string, optional): request timeout, default 30s
• include_messages (bool, optional): include conversation history, default true
• include_tool_calls (bool, optional): include tool call records, default false
• criteria (string, optional): evaluation criteria forwarded in request
• min_score (float64, optional): minimum score threshold
• extra (map[string]any, optional): arbitrary data forwarded in request

func (h *RestEvalHandler) Eval(
	ctx context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *RestEvalHandler) Type() string

type RestEvalSessionHandler ¶

type RestEvalSessionHandler struct{}

RestEvalSessionHandler evaluates an entire conversation by POSTing all assistant messages to an external HTTP endpoint.

Params: same as RestEvalHandler.

func (h *RestEvalSessionHandler) Eval(
	ctx context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *RestEvalSessionHandler) Type() string

type RoleViolationHandler ¶

type RoleViolationHandler struct{}

RoleViolationHandler scores whether the assistant output breaks its assigned role, persona, or instruction set. Equivalent in name to DeepEval `role_violation`. Default wiring is as a guardrail; scenarios observe firing via `guardrail_triggered`.

The handler injects the active system prompt (if available via evalCtx.Metadata["system_prompt"]) into the judge prompt so the judge can decide whether the answer deviates from it. If no system prompt is supplied via metadata or params, the judge falls back to generic role-consistency scoring.

Default prompts adapted from the public DeepEval reference implementation (Apache 2.0).

Params (all optional):

• system_prompt (string): the role / persona the answer should adhere to; overrides metadata. Distinct from the standard llm_judge `system_prompt` which controls the JUDGE's prompt.
• rubric, model, criteria: standard llm_judge knobs

func (h *RoleViolationHandler) Eval(
	ctx context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *RoleViolationHandler) Type() string

type SentenceCountHandler ¶

type SentenceCountHandler struct{}

SentenceCountHandler counts sentences in the current output. This is a pure measurement handler — it always passes. Params: (none required — returns count as measurement)

func (h *SentenceCountHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	_ map[string]any,
) (*evals.EvalResult, error)

func (h *SentenceCountHandler) Type() string

type SkillActivatedHandler ¶

type SkillActivatedHandler struct{}

SkillActivatedHandler checks that specific skills were activated. Scans evalCtx.ToolCalls for "skill__activate" calls and extracts the "name" argument. Params: skill_names []string, min_calls int (optional, default 1).

func (h *SkillActivatedHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *SkillActivatedHandler) Type() string

type SkillActivationOrderHandler ¶

type SkillActivationOrderHandler struct{}

SkillActivationOrderHandler checks that skills were activated in a specified subsequence order. Params: sequence []string — the expected skill names in order.

func (h *SkillActivationOrderHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *SkillActivationOrderHandler) Type() string

type SkillNotActivatedHandler ¶

type SkillNotActivatedHandler struct{}

SkillNotActivatedHandler checks that specific skills were NOT activated. Params: skill_names []string.

func (h *SkillNotActivatedHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *SkillNotActivatedHandler) Type() string

type SpecJudgeProvider ¶

type SpecJudgeProvider struct {
	// contains filtered or unexported fields
}

SpecJudgeProvider implements JudgeProvider by creating a provider from a ProviderSpec. This is the standard implementation used by Arena and any caller that has judge targets as ProviderSpecs.

func NewSpecJudgeProvider(spec *providers.ProviderSpec) *SpecJudgeProvider

func (sp *SpecJudgeProvider) Judge(ctx context.Context, opts JudgeOpts) (*JudgeResult, error)

type TextSentimentHandler ¶

type TextSentimentHandler struct{}

TextSentimentHandler is a pure eval primitive: it scores text against a sentiment classification model (e.g. `cardiffnlp/twitter-roberta-base-sentiment-latest`, `distilbert-base-uncased-finetuned-sst-2-english`) and emits the score for the chosen expected_label.

Threshold judgment lives on the `type: assertion` wrapper:

• type: assertion params: eval_type: text_sentiment eval_params: model: cardiffnlp/twitter-roberta-base-sentiment-latest expected_label: positive min_score: 0.7

Pack-level runtime eval (emits raw signal, no judgment):

evals:
  - id: response-sentiment
    type: text_sentiment
    trigger: every_turn
    params:
      model: cardiffnlp/twitter-roberta-base-sentiment-latest
      expected_label: positive

Params:

• model string (required)
• expected_label string (required)
• message_role string (optional, default "assistant")
• message_index int (optional, default -1)
• classifier_id string (optional)

func (h *TextSentimentHandler) Eval(
	ctx context.Context, evalCtx *evals.EvalContext, params map[string]any,
) (*evals.EvalResult, error)

func (h *TextSentimentHandler) Type() string

type TextToxicityHandler ¶

type TextToxicityHandler struct{}

TextToxicityHandler is a pure eval primitive: it scores text against a toxicity classification model (e.g. `unitary/toxic-bert`, `s-nlp/roberta_toxicity_classifier`) and emits the score for the chosen expected_label. Distinct from the legacy `toxicity` handler, which is an LLM-judge with the same name — `text_toxicity` is the deterministic classifier path that depends on `classify.TextClassifier` and an `inference` provider.

Threshold judgment lives on the `type: assertion` wrapper:

# "this output should NOT be toxic"
- type: assertion
  params:
    eval_type: text_toxicity
    eval_params: { model: unitary/toxic-bert, expected_label: toxic }
    max_score: 0.3

# "this output should sit in the neutral class"
- type: assertion
  params:
    eval_type: text_toxicity
    eval_params: { model: s-nlp/roberta_toxicity_classifier, expected_label: neutral }
    min_score: 0.7

Pack-level runtime eval (emits the raw signal, no judgment):

evals:
  - id: response-toxicity
    type: text_toxicity
    trigger: every_turn
    params: { model: unitary/toxic-bert, expected_label: toxic }

Params:

• model string (required) — backend model id
• expected_label string (required) — label whose score is emitted
• message_role string (optional, default "assistant")
• message_index int (optional, default -1)
• classifier_id string (optional)

func (h *TextToxicityHandler) Eval(
	ctx context.Context, evalCtx *evals.EvalContext, params map[string]any,
) (*evals.EvalResult, error)

func (h *TextToxicityHandler) Type() string

type ToolAntiPatternHandler ¶

type ToolAntiPatternHandler struct{}

ToolAntiPatternHandler checks that tool calls do NOT contain forbidden subsequences. Params: patterns []map[string]any — each with "sequence" ([]string) and optional "message" (string).

func (h *ToolAntiPatternHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *ToolAntiPatternHandler) Type() string

type ToolArgsExcludedSessionHandler ¶

type ToolArgsExcludedSessionHandler struct{}

ToolArgsExcludedSessionHandler checks that a tool was NOT called with specific argument values across the session. Params: tool_name string, excluded_args map[string]any. Also accepts legacy param forbidden_args map[string][]any (from the original tools_not_called_with_args validator).

func (h *ToolArgsExcludedSessionHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (_ *evals.EvalResult, _ error)

func (h *ToolArgsExcludedSessionHandler) Type() string

type ToolArgsHandler ¶

type ToolArgsHandler struct{}

ToolArgsHandler checks that a tool was called with specific args. Params: tool_name string, expected_args map[string]any.

func (h *ToolArgsHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (result *evals.EvalResult, err error)

func (h *ToolArgsHandler) Type() string

type ToolArgsSessionHandler ¶

type ToolArgsSessionHandler struct{}

ToolArgsSessionHandler checks that a tool was called with specific arguments across the session. Params: tool_name string, expected_args map[string]any.

func (h *ToolArgsSessionHandler) Eval(
	ctx context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (_ *evals.EvalResult, _ error)

func (h *ToolArgsSessionHandler) Type() string

type ToolCallChainHandler ¶

type ToolCallChainHandler struct{}

ToolCallChainHandler checks a dependency chain of tool calls with per-step constraints. Params: steps []map — each with tool, result_includes, result_matches, args_match, no_error.

func (h *ToolCallChainHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *ToolCallChainHandler) Type() string

type ToolCallCountHandler ¶

type ToolCallCountHandler struct{}

ToolCallCountHandler checks the count of tool calls within bounds. Params: tool string (optional), min int (optional), max int (optional).

func (h *ToolCallCountHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *ToolCallCountHandler) Type() string

type ToolCallSequenceHandler ¶

type ToolCallSequenceHandler struct{}

ToolCallSequenceHandler checks that tool calls appear in a specified subsequence order. Params: sequence []string — the expected tool names in order.

func (h *ToolCallSequenceHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *ToolCallSequenceHandler) Type() string

type ToolCallsWithArgsHandler ¶

type ToolCallsWithArgsHandler struct{}

ToolCallsWithArgsHandler checks that a tool was called with expected arguments. Supports exact value matching (expected_args), regex pattern matching (args_match), and result-level constraints (result_includes, result_matches, no_error).

func (h *ToolCallsWithArgsHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *ToolCallsWithArgsHandler) Type() string

type ToolEfficiencyHandler ¶

type ToolEfficiencyHandler struct{}

ToolEfficiencyHandler checks tool usage efficiency metrics. Params:

• max_calls: int — maximum total tool calls allowed (optional)
• max_errors: int — maximum tool errors allowed (optional)
• max_error_rate: float64 — maximum error rate 0.0-1.0 (optional)

func (h *ToolEfficiencyHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *ToolEfficiencyHandler) Type() string

type ToolExecHandler ¶

type ToolExecHandler struct{}

ToolExecHandler invokes a tool by name through the runtime tool registry and asserts the call succeeded. The pass condition is:

• tools.Registry.Execute returns no error, AND
• the resulting ToolResult has an empty Error field.

This makes it a generic "is this tool happy" gate that works with any registered tool: MCP-backed (e.g. a sandbox's run_tests), HTTP/local executors, custom client tools — whatever the host has wired up. The handler doesn't know or care about the transport.

Params:

• tool string (required) — registry name of the tool to invoke
• args map[string]any (optional) — arguments passed verbatim to Execute
• timeout_seconds int (optional) — bounds the call; default 120

Score is 1.0 on success, 0.0 on any failure (error returned, result has Error field set, missing tool, missing registry, timeout).

func (h *ToolExecHandler) Eval(
	ctx context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *ToolExecHandler) Type() string

type ToolNoRepeatHandler ¶

type ToolNoRepeatHandler struct{}

ToolNoRepeatHandler detects consecutive repeated calls to the same tool. Params:

• tools: []string — tool names to check (empty = all tools)
• max_repeats: int — maximum allowed consecutive calls to the same tool (default 1)

func (h *ToolNoRepeatHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *ToolNoRepeatHandler) Type() string

type ToolResultHasMediaHandler ¶

type ToolResultHasMediaHandler struct{}

ToolResultHasMediaHandler asserts that a named tool call returned media content of a given type. Params: tool string, media_type string ("image", "audio", "video", "document").

func (h *ToolResultHasMediaHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *ToolResultHasMediaHandler) Type() string

type ToolResultIncludesHandler ¶

type ToolResultIncludesHandler struct{}

ToolResultIncludesHandler checks that tool results contain expected substrings. Params: tool string, patterns []string, occurrence int (optional, default 1).

func (h *ToolResultIncludesHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *ToolResultIncludesHandler) Type() string

type ToolResultMatchesHandler ¶

type ToolResultMatchesHandler struct{}

ToolResultMatchesHandler checks that tool results match a regex pattern. Params: tool string, pattern string, occurrence int (optional, default 1).

func (h *ToolResultMatchesHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *ToolResultMatchesHandler) Type() string

type ToolResultMediaTypeHandler ¶

type ToolResultMediaTypeHandler struct{}

ToolResultMediaTypeHandler asserts the MIME type of media in a tool result. Params: tool string, mime_type string.

func (h *ToolResultMediaTypeHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *ToolResultMediaTypeHandler) Type() string

type ToolsCalledHandler ¶

type ToolsCalledHandler struct{}

ToolsCalledHandler checks if specific tools were called successfully.

By default, only counts tool calls that completed without error. Use ignore_validation: true to also count calls that failed argument validation.

Params:

• tool_names/tools []string — required tool names
• min_calls int — minimum calls per tool (default 1)
• max_calls int — maximum calls per tool (default unbounded, use 0 to forbid). When any listed tool exceeds max_calls the assertion fails hard with score 0, regardless of min_calls satisfaction.
• ignore_validation bool — count validation failures as successful (default false)
• require_args bool — only count calls with non-empty arguments (default false)

func (h *ToolsCalledHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (result *evals.EvalResult, err error)

func (h *ToolsCalledHandler) Type() string

type ToolsCalledSessionHandler ¶

type ToolsCalledSessionHandler struct{}

ToolsCalledSessionHandler checks that specific tools were called across the full session. Params: tool_names []string, min_calls int (optional, default 1).

func (h *ToolsCalledSessionHandler) Eval(
	ctx context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (_ *evals.EvalResult, _ error)

func (h *ToolsCalledSessionHandler) Type() string

type ToolsNotCalledHandler ¶

type ToolsNotCalledHandler struct{}

ToolsNotCalledHandler checks that specific tools were NOT called. Params: tool_names []string.

func (h *ToolsNotCalledHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (result *evals.EvalResult, err error)

func (h *ToolsNotCalledHandler) Type() string

type ToolsNotCalledSessionHandler ¶

type ToolsNotCalledSessionHandler struct{}

ToolsNotCalledSessionHandler checks that specific tools were NOT called anywhere in the session. Params: tool_names []string.

func (h *ToolsNotCalledSessionHandler) Eval(
	ctx context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (_ *evals.EvalResult, _ error)

func (h *ToolsNotCalledSessionHandler) Type() string

type TopicPolicyHandler ¶ added in v2.2.0

type TopicPolicyHandler struct {
	// contains filtered or unexported fields
}

TopicPolicyHandler confines a conversation to a declared subject scope, decided by a classify.TopicClassifier rather than by the model being governed. Declared as a guardrail, it gates input by default and a denied turn never reaches the primary provider.

Pack validator (the normal declaration site):

validators:
  - type: topic_policy
    message: "I can help with questions about AltairaLabs products."
    params:
      description: Helps users evaluate and operate AltairaLabs products.
      allowed: [Omnia and PromptKit, licensing and support]
      disallowed: [politics]

Host side, an ordinary provider file:

id: topic-control
role: inference
type: nvidia-topic-control
base_url: http://topic-control:8000/v1

Params: description (required), allowed (required, non-empty), disallowed, small_talk (allow|deny), examples.{allowed,disallowed}, on_deny (block|respond), on_unknown (deny|allow), on_error (deny|allow), recent_turns (>= 0), classifier_id.

func (h *TopicPolicyHandler) Eval(
	ctx context.Context, evalCtx *evals.EvalContext, params map[string]any,
) (*evals.EvalResult, error)

func (h *TopicPolicyHandler) Type() string

func (h *TopicPolicyHandler) ValidateParams(params map[string]any) error

type ToxicityHandler ¶

type ToxicityHandler struct{}

ToxicityHandler scores the assistant output for insults, harassment, threats, hate speech via the LLM judge. DeepEval-equivalent name; distinct from the classifier-backed text_toxicity. Default-wired as a guardrail. Standard llm_judge params.

func (h *ToxicityHandler) Eval(
	ctx context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *ToxicityHandler) Type() string

type VideoDurationHandler ¶

type VideoDurationHandler struct{}

VideoDurationHandler checks that video duration is within range. Params: min_seconds float64, max_seconds float64.

func (h *VideoDurationHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *VideoDurationHandler) Type() string

type VideoResolutionHandler ¶

type VideoResolutionHandler struct{}

VideoResolutionHandler checks that video resolution meets requirements. Params: min_width, max_width, min_height, max_height, presets []string.

func (h *VideoResolutionHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *VideoResolutionHandler) Type() string

type WorkflowCompleteHandler ¶

type WorkflowCompleteHandler struct{}

WorkflowCompleteHandler checks that the workflow reached a terminal state. Reads evalCtx.Extras["workflow_complete"] (bool) and ["workflow_current_state"] (string).

func (h *WorkflowCompleteHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	_ map[string]any,
) (*evals.EvalResult, error)

func (h *WorkflowCompleteHandler) Type() string

type WorkflowSpokeInStateHandler ¶

type WorkflowSpokeInStateHandler struct{}

WorkflowSpokeInStateHandler checks that a workflow state actually produced assistant output, rather than merely being entered.

This closes the blind spot that let the in-turn handoff bug ship. state_is, transitioned_to, workflow_complete and workflow_transition_order all read the state machine's history, so a transition that generates no output at all satisfies every one of them — which is exactly what was broken: the machine advanced and the destination state never spoke.

Params: state string (required).

func (h *WorkflowSpokeInStateHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *WorkflowSpokeInStateHandler) Type() string

func (h *WorkflowSpokeInStateHandler) ValidateParams(params map[string]any) error

type WorkflowStateIsHandler ¶

type WorkflowStateIsHandler struct{}

WorkflowStateIsHandler checks that the current workflow state matches an expected value. Params: state string (required).

func (h *WorkflowStateIsHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *WorkflowStateIsHandler) Type() string

func (h *WorkflowStateIsHandler) ValidateParams(params map[string]any) error

type WorkflowToolAccessHandler ¶

type WorkflowToolAccessHandler struct{}

WorkflowToolAccessHandler enforces that tools are only called when the workflow is in a state that permits them. Params: rules []map[string]any — each with "state" (string) and "allowed" ([]string).

func (h *WorkflowToolAccessHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *WorkflowToolAccessHandler) Type() string

type WorkflowTransitionOrderHandler ¶

type WorkflowTransitionOrderHandler struct{}

WorkflowTransitionOrderHandler checks that workflow state transitions happen in an expected order. Params: sequence []string (required) — expected ordered list of states that must appear as a subsequence.

func (h *WorkflowTransitionOrderHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *WorkflowTransitionOrderHandler) Type() string

type WorkflowTransitionedToHandler ¶

type WorkflowTransitionedToHandler struct{}

WorkflowTransitionedToHandler checks that a transition to a specific state occurred. Params: state string (required).

func (h *WorkflowTransitionedToHandler) Eval(
	_ context.Context,
	evalCtx *evals.EvalContext,
	params map[string]any,
) (*evals.EvalResult, error)

func (h *WorkflowTransitionedToHandler) Type() string