predicates

package
v0.32.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 1, 2026 License: MIT Imports: 14 Imported by: 0

Documentation

Overview

Package predicates implements the ICM workflow predicate evaluator.

The evaluator dispatches Predicate values produced by the workspace loader against an artifact (the bytes a stage emitted) and returns a Result per predicate. Six predicate types are supported: schema, regex, native, command, llm, human. Each type has a dedicated per-file evaluator in this package.

Schema, regex, and command are implemented end-to-end. Native dispatches to a process-local handler registry; an unregistered handler reports a non-fatal failure result. LLM and human predicates invoke pluggable dispatchers (JudgeDispatch / HumanDispatch); when no dispatcher is configured, evaluation returns a failure result with an explanatory feedback string. Wiring those dispatchers happens in later commits.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func PredicateSchemaName

func PredicateSchemaName(instanceID, stageID, predName string) string

PredicateSchemaName returns the canonical schema-registry name for a predicate. The icm plugin uses this helper at Ready() to register every output schema; the evaluator uses it at lookup time. Both halves stay in lockstep by delegating to the icmtypes canonical helper.

Format: "icm.[<instance suffix>.]<stageID>.<predName>". When predName is "output" (the synthesized stage-output schema predicate) the result is the same as StageOutputSchemaName so output schemas and named schema predicates share one namespace.

Types

type Evaluator

type Evaluator struct {
	// Schemas is the engine schema registry consulted for `type:
	// schema` predicates. Required; schema predicates without a
	// configured registry return a failure result.
	Schemas *engine.SchemaRegistry
	// Sandbox executes `type: command` predicates. Required for command
	// predicates; absent backend yields a failure result.
	Sandbox sandbox.Sandbox
	// Bus, when non-nil, receives an `icm.predicate.failed` event for
	// every failing predicate. Tests can leave this nil to skip
	// emission.
	Bus engine.EventBus
	// Logger records evaluation events. Must be non-nil.
	Logger *slog.Logger
	// Judge dispatches `type: llm` predicates. Nil until wired by the
	// plugin (step 6); evaluator returns "judge not configured" until
	// then.
	Judge JudgeDispatch
	// Human dispatches `type: human` predicates. Nil until wired by the
	// plugin (step 6); evaluator returns "human handler not configured"
	// until then.
	Human HumanDispatch
	// CommandTimeoutSecs is the plugin-config default applied to
	// command predicates whose own TimeoutSeconds and whose
	// StageBudgetTimeoutSec are both zero. Falls back to 30s when this
	// is also zero.
	CommandTimeoutSecs int
	// contains filtered or unexported fields
}

Evaluator runs Predicate values against artifacts. Construct via NewEvaluator and configure pluggable dispatchers via the public fields before use.

func NewEvaluator

func NewEvaluator(schemas *engine.SchemaRegistry, sb sandbox.Sandbox, bus engine.EventBus, logger *slog.Logger) *Evaluator

NewEvaluator constructs an Evaluator with the required dependencies. Dispatchers (Judge, Human) and config knobs (CommandTimeoutSecs) can be set directly on the returned value before use.

func (*Evaluator) Evaluate

func (e *Evaluator) Evaluate(ctx context.Context, p *workspace.Predicate, artifact []byte, sc StageEvalContext) Result

Evaluate runs a single predicate. The returned Result always has its Name, Type, and Elapsed fields populated regardless of outcome.

func (*Evaluator) EvaluateAll

func (e *Evaluator) EvaluateAll(ctx context.Context, ps []workspace.Predicate, artifact []byte, sc StageEvalContext) (allPassed bool, results []Result)

EvaluateAll runs predicates in declared order, short-circuiting on the first failure. Returns allPassed=true only when every predicate returned Verdict=true. When Bus is non-nil, every failing predicate emits an icm.predicate.failed event.

func (*Evaluator) LookupNative

func (e *Evaluator) LookupNative(name string) (NativeHandler, bool)

LookupNative returns the registered handler for name, if any.

func (*Evaluator) RegisterNative

func (e *Evaluator) RegisterNative(name string, h NativeHandler)

RegisterNative records h as the handler for name. Subsequent registrations under the same name replace the prior handler.

type HumanDispatch

type HumanDispatch func(ctx context.Context, p *workspace.Predicate, artifact []byte, sc StageEvalContext) (verdict bool, feedback string, err error)

HumanDispatch is invoked by human predicates. The dispatcher prompts the operator (typically via the HITL plugin) and returns their verdict. Returning a non-nil error is treated as a failure result; the error message becomes the feedback.

type JudgeDispatch

type JudgeDispatch func(ctx context.Context, p *workspace.Predicate, artifact []byte, sc StageEvalContext) (verdict bool, feedback string, score *float64, err error)

JudgeDispatch is invoked by llm predicates. The dispatcher calls whichever judge model the predicate (or workspace default) requests and returns the parsed verdict. Returning a non-nil error becomes a failing result with the error message as feedback.

type NativeHandler

type NativeHandler interface {
	// Evaluate runs the handler against the given artifact and returns
	// its outcome. Handlers should be cheap and non-blocking — long
	// work belongs in command or llm predicates.
	Evaluate(ctx context.Context, args map[string]any, artifact []byte) NativeResult
}

NativeHandler is the contract for in-process predicate handlers registered via Evaluator.RegisterNative. Step 5 of the ICM plugin build wires the real handlers; the skeleton ships an empty registry.

type NativeResult

type NativeResult struct {
	Verdict  bool
	Feedback string
	Score    *float64
}

NativeResult is what NativeHandler implementations return. The evaluator copies these fields into a Result and overwrites Name/Type with the predicate's declared values so handlers cannot accidentally rename themselves in event payloads.

type Result

type Result struct {
	// Name mirrors the predicate's Name field for stable identification
	// in event payloads and feedback strings.
	Name string
	// Type echoes the predicate's discriminator so callers can route
	// results without re-reading the original Predicate.
	Type workspace.PredicateType
	// Verdict is true on pass, false on fail.
	Verdict bool
	// Feedback explains a failure (or, optionally, a pass). Empty
	// strings on failing results are still legal; downstream code falls
	// back to a generic message.
	Feedback string
	// Score is an optional 0..1 confidence number set by LLM judges.
	Score *float64
	// Elapsed is the wall time the evaluator spent on this predicate.
	Elapsed time.Duration
}

Result is the unified outcome of evaluating a single predicate. Verdict=true means the artifact satisfied the predicate; Verdict=false means it did not, in which case Feedback should describe why.

type StageEvalContext

type StageEvalContext struct {
	// RunID is the run that owns this evaluation.
	RunID string
	// StageID is the stage being evaluated.
	StageID string
	// ItemID names the fan-out item, when applicable.
	ItemID string
	// Iteration is the 1-based loop iteration index, when applicable.
	Iteration int
	// Container labels what surface invoked the evaluator. Standard
	// values: "output.validators", "loop.until", "verifier".
	Container string
	// ParentTurnID is the dispatching turn's ID, propagated for
	// causation tracking by downstream predicate types that surface to
	// LLM judges or human handlers.
	ParentTurnID string
	// ParentDepth is the sub-agent depth of the dispatching turn.
	ParentDepth int
	// WorkspaceRoot is the absolute root path of the workspace, used by
	// command predicates to resolve relative script paths.
	WorkspaceRoot string
	// StageBudgetTimeoutSec is the stage's per-call timeout budget in
	// seconds (0 = no stage-level override). Command predicates consult
	// this when their own TimeoutSeconds is zero.
	StageBudgetTimeoutSec int
	// InstanceID names the icm plugin instance owning this run.
	// PredicateSchemaName uses it to derive the schema-registry name.
	InstanceID string
}

StageEvalContext carries the per-stage values the evaluator needs to dispatch certain predicate types. Built by the orchestrator once per stage invocation and passed through to every Evaluate call for that stage.

Directories

Path Synopsis
Package builtins ships the four baked-in native predicate handlers for the ICM workflow plugin: word_count_under, word_count_over, contains_required_ids, and json_path_exists.
Package builtins ships the four baked-in native predicate handlers for the ICM workflow plugin: word_count_under, word_count_over, contains_required_ids, and json_path_exists.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL