stages

package
v1.5.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 26, 2026 License: Apache-2.0 Imports: 16 Imported by: 0

Documentation

Overview

Package stages provides arena-specific pipeline stages for test execution.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type ArenaAssertionStage

type ArenaAssertionStage struct {
	stage.BaseStage
	// contains filtered or unexported fields
}

ArenaAssertionStage validates assertions after LLM responses.

func NewArenaAssertionStage

func NewArenaAssertionStage(
	assertionConfigs []assertions.AssertionConfig,
) *ArenaAssertionStage

NewArenaAssertionStage creates a new assertion stage.

func (*ArenaAssertionStage) Process

func (s *ArenaAssertionStage) Process(
	ctx context.Context,
	input <-chan stage.StreamElement,
	output chan<- stage.StreamElement,
) error

Process validates assertions on the stream elements.

func (*ArenaAssertionStage) WithTurnEvalRunner added in v1.3.2

func (s *ArenaAssertionStage) WithTurnEvalRunner(runner TurnEvalRunner, sessionID string) *ArenaAssertionStage

WithTurnEvalRunner sets the eval runner for assertion execution.

type ArenaStateStoreSaveStage

type ArenaStateStoreSaveStage struct {
	stage.BaseStage
	// contains filtered or unexported fields
}

ArenaStateStoreSaveStage saves conversation state with telemetry to ArenaStateStore. This stage captures validation results, turn metrics, and cost information for Arena testing and analysis.

func NewArenaStateStoreSaveStage

func NewArenaStateStoreSaveStage(config *pipeline.StateStoreConfig) *ArenaStateStoreSaveStage

NewArenaStateStoreSaveStage creates a new Arena state store save stage.

func NewArenaStateStoreSaveStageWithTurnState added in v1.4.7

func NewArenaStateStoreSaveStageWithTurnState(
	config *pipeline.StateStoreConfig, turnState *stage.TurnState,
) *ArenaStateStoreSaveStage

NewArenaStateStoreSaveStageWithTurnState creates an Arena state store save stage that reads the rendered system prompt from the supplied TurnState. Falls back to config.Metadata["system_prompt"] when the TurnState is empty (used by tests that wire the prompt via config).

func (*ArenaStateStoreSaveStage) Process

func (s *ArenaStateStoreSaveStage) Process(
	ctx context.Context, input <-chan stage.StreamElement, output chan<- stage.StreamElement,
) error

Process collects messages and saves them incrementally to Arena state store. Messages are saved after each turn completion (when an element contains a Message). This ensures conversation state is captured in real-time as turns complete.

The conversation state is loaded lazily on the first save and reused for subsequent saves within the same Process call — avoiding O(N) Loads on scenarios with many Message elements.

func (*ArenaStateStoreSaveStage) WithEmitter added in v1.4.7

WithEmitter wires an events.Emitter so the stage broadcasts each message to the event bus the moment it arrives. Live UIs (TUI conversation panel, web SSE relay) can then render turns as they happen instead of waiting for run completion. Stage-level state save remains the source of truth for replay and post-run results.

type AssistantTTSFilterStage added in v1.5.3

type AssistantTTSFilterStage struct {
	stage.BaseStage
}

AssistantTTSFilterStage sits between the ArenaStateStoreSaveStage and the TTSStageWithInterruption in the VAD composed pipeline. It drops any Message element whose Role is not "assistant", preventing the user's own transcript (wrapped as a user Message by STTUserMessageStage) from being spoken back through TTS.

All non-Message elements — audio, text, EndOfStream, and control elements — are forwarded unchanged so the TTS stage receives everything it needs.

This is a Filter stage: it passes through assistant Messages and non-Message elements, and drops user/system/tool Messages.

func NewAssistantTTSFilterStage added in v1.5.3

func NewAssistantTTSFilterStage() *AssistantTTSFilterStage

NewAssistantTTSFilterStage creates a filter that only forwards assistant Message elements (and all non-Message elements) to the TTS stage.

func (*AssistantTTSFilterStage) Process added in v1.5.3

func (s *AssistantTTSFilterStage) Process(
	ctx context.Context,
	input <-chan stage.StreamElement,
	output chan<- stage.StreamElement,
) error

Process forwards every element unchanged except Message elements whose Role is not "assistant" — those are silently dropped.

type CompletionInstructionStage added in v1.3.16

type CompletionInstructionStage struct {
	stage.BaseStage
	// contains filtered or unexported fields
}

CompletionInstructionStage appends natural termination instructions to the system prompt assembled by PersonaAssemblyStage. The instruction is appended exactly once per Turn — TurnState.SystemPrompt is the source of truth and the stage is idempotent across the elements that flow through it.

func NewCompletionInstructionStageWithTurnState added in v1.4.7

func NewCompletionInstructionStageWithTurnState(
	instruction string, turnState *stage.TurnState,
) *CompletionInstructionStage

NewCompletionInstructionStageWithTurnState creates a stage that reads and writes the system prompt through the shared *TurnState.

func (*CompletionInstructionStage) Process added in v1.3.16

func (s *CompletionInstructionStage) Process(ctx context.Context, input <-chan stage.StreamElement, output chan<- stage.StreamElement) error

Process appends the completion instruction to the system prompt.

type HistoryInjectionStage

type HistoryInjectionStage struct {
	stage.BaseStage
	// contains filtered or unexported fields
}

HistoryInjectionStage prepends conversation history to the pipeline stream. This is useful for self-play scenarios where history needs to be explicitly provided before the LLM generates the next user message.

The stage emits all history messages first, then forwards any incoming elements.

When swapRoles is true, user↔assistant roles are swapped so that the self-play LLM sees its own prior outputs as "assistant" and the target's responses as "user". This prevents the self-play LLM from confusing itself with the target assistant.

func NewHistoryInjectionStage

func NewHistoryInjectionStage(history []types.Message) *HistoryInjectionStage

NewHistoryInjectionStage creates a new history injection stage.

func NewHistoryInjectionStageSwapped added in v1.3.16

func NewHistoryInjectionStageSwapped(history []types.Message) *HistoryInjectionStage

NewHistoryInjectionStageSwapped creates a history injection stage that swaps user↔assistant roles. Use this for self-play generation so the LLM sees the conversation from its own perspective (its prior outputs as "assistant").

func (*HistoryInjectionStage) Process

func (s *HistoryInjectionStage) Process(ctx context.Context, input <-chan stage.StreamElement, output chan<- stage.StreamElement) error

Process emits history messages first, then forwards all incoming elements.

type MockScenarioContextStage

type MockScenarioContextStage struct {
	stage.BaseStage
	// contains filtered or unexported fields
}

MockScenarioContextStage adds scenario context to TurnState's ProviderRequestMetadata so MockProvider can select scenario-specific canned responses.

This stage should be placed before ProviderStage in the pipeline when using MockProvider to ensure scenario context is available.

func NewMockScenarioContextStageWithTurnState added in v1.4.7

func NewMockScenarioContextStageWithTurnState(
	scenario *config.Scenario, turnState *stage.TurnState,
) *MockScenarioContextStage

NewMockScenarioContextStageWithTurnState creates a stage that writes scenario context into the shared *TurnState's ProviderRequestMetadata.

func (*MockScenarioContextStage) Process

func (s *MockScenarioContextStage) Process(ctx context.Context, input <-chan stage.StreamElement, output chan<- stage.StreamElement) error

Process writes scenario context into TurnState and forwards all elements unchanged.

type PersonaAssemblyStage

type PersonaAssemblyStage struct {
	stage.BaseStage
	// contains filtered or unexported fields
}

PersonaAssemblyStage assembles persona prompts using the same fragment/template system as PromptAssemblyStage.

This stage mirrors the behavior of PromptAssemblyMiddleware but for personas:

  • Uses persona's BuildSystemPrompt() which handles fragment assembly
  • Supports template variable substitution with {{variable}} syntax
  • Injects persona-specific variables (goals, constraints, style)
  • Optionally prepends a TTS characterization rubric when the persona has opted in via style.expressive (see issue #1130)
  • Writes the rendered system prompt and base variables into TurnState

func NewPersonaAssemblyStageWithTurnState added in v1.4.7

func NewPersonaAssemblyStageWithTurnState(
	persona *config.UserPersonaPack,
	region string,
	baseVariables map[string]string,
	turnState *stage.TurnState,
) *PersonaAssemblyStage

NewPersonaAssemblyStageWithTurnState creates a persona assembly stage that writes the rendered persona system prompt into the shared *TurnState.

func (*PersonaAssemblyStage) Process

func (s *PersonaAssemblyStage) Process(ctx context.Context, input <-chan stage.StreamElement, output chan<- stage.StreamElement) error

Process assembles the persona prompt and writes it into TurnState. All input elements are forwarded unchanged.

func (*PersonaAssemblyStage) WithProviderRubric added in v1.4.7

func (s *PersonaAssemblyStage) WithProviderRubric(rubric string) *PersonaAssemblyStage

WithProviderRubric sets the TTS provider's characterization rubric. The stage uses it as the default when the persona opts into expressive output (style.expressive) and does not provide its own override. Empty string disables the default and is the no-op path for non-expressive providers.

type STTUserMessageStage added in v1.5.3

type STTUserMessageStage struct {
	stage.BaseStage
}

STTUserMessageStage converts a transcribed text element (as produced by the STT stage) into a user Message element so the downstream prompt-assembly, provider, and state-store-save stages treat the spoken turn exactly like a typed user message. This is the bridge that makes the composed VAD voice pipeline materialize history identically to a text run: ProviderStage only accumulates Message elements, so a bare Text element from STT would otherwise never reach the provider or be persisted.

Text-only elements are wrapped; elements that already carry a Message, plus audio / EndOfStream / non-text elements, are forwarded unchanged.

This is a Transform stage: text element -> user message element (1:1).

func NewSTTUserMessageStage added in v1.5.3

func NewSTTUserMessageStage() *STTUserMessageStage

NewSTTUserMessageStage creates a stage that wraps STT text into user messages.

func (*STTUserMessageStage) Process added in v1.5.3

func (s *STTUserMessageStage) Process(
	ctx context.Context,
	input <-chan stage.StreamElement,
	output chan<- stage.StreamElement,
) error

Process wraps text elements into user message elements and forwards the rest.

type ScenarioContextExtractionStage

type ScenarioContextExtractionStage struct {
	stage.BaseStage
	// contains filtered or unexported fields
}

ScenarioContextExtractionStage extracts context using scenario metadata and conversation history. This is designed for Arena use where rich scenario metadata is available.

When scenario metadata is present, it uses: - Scenario metadata variables (domain, user role from scenario definition) - Scenario description and task type - Message analysis as fallback

Extracted variables are merged into TurnState.Variables, where TemplateStage reads them when rendering the system prompt.

func NewScenarioContextExtractionStageWithTurnState added in v1.4.7

func NewScenarioContextExtractionStageWithTurnState(
	scenario *config.Scenario, turnState *stage.TurnState,
) *ScenarioContextExtractionStage

NewScenarioContextExtractionStageWithTurnState creates a scenario context extraction stage that writes extracted variables into the shared *TurnState.

func (*ScenarioContextExtractionStage) Process

func (s *ScenarioContextExtractionStage) Process(ctx context.Context, input <-chan stage.StreamElement, output chan<- stage.StreamElement) error

Process extracts scenario context and writes it to TurnState.Variables.

type SelfPlayUserTurnContextStage

type SelfPlayUserTurnContextStage struct {
	stage.BaseStage
	// contains filtered or unexported fields
}

SelfPlayUserTurnContextStage adds scenario context for the NEXT user turn (completed user turns + 1). Intended only for self-play user generation.

This stage writes scenario/turn coordination keys into TurnState's ProviderRequestMetadata so the mock provider can select the appropriate canned response by scenario id, turn number, and persona.

func NewSelfPlayUserTurnContextStageWithHintAndTurnState added in v1.4.7

func NewSelfPlayUserTurnContextStageWithHintAndTurnState(
	scenario *config.Scenario,
	turnIndexHint int,
	personaID string,
	turnState *stage.TurnState,
) *SelfPlayUserTurnContextStage

NewSelfPlayUserTurnContextStageWithHintAndTurnState creates a self-play context stage with an explicit turn index. The turnIndexHint should be the 1-indexed selfplay turn number (first selfplay = 1). Used when the scenario has mixed file-based and selfplay turns.

func NewSelfPlayUserTurnContextStageWithTurnState added in v1.4.7

func NewSelfPlayUserTurnContextStageWithTurnState(
	scenario *config.Scenario,
	personaID string,
	turnState *stage.TurnState,
) *SelfPlayUserTurnContextStage

NewSelfPlayUserTurnContextStageWithTurnState creates a self-play context stage that writes coordination metadata into the shared *TurnState.

func (*SelfPlayUserTurnContextStage) Process

func (s *SelfPlayUserTurnContextStage) Process(ctx context.Context, input <-chan stage.StreamElement, output chan<- stage.StreamElement) error

Process writes next-turn self-play context into TurnState's provider request metadata, then forwards all input elements unchanged.

type SkillInstructionStage added in v1.4.5

type SkillInstructionStage struct {
	stage.BaseStage
	// contains filtered or unexported fields
}

SkillInstructionStage appends preloaded skill instructions to the system prompt so the model sees skills marked preload: true from turn 1 without having to call skill__activate. The instructions are appended exactly once per Turn into TurnState.SystemPrompt.

func NewSkillInstructionStageWithTurnState added in v1.4.7

func NewSkillInstructionStageWithTurnState(
	instructions string, turnState *stage.TurnState,
) *SkillInstructionStage

NewSkillInstructionStageWithTurnState creates a stage that reads and writes the system prompt through the shared *TurnState.

func (*SkillInstructionStage) Process added in v1.4.5

func (s *SkillInstructionStage) Process(ctx context.Context, input <-chan stage.StreamElement, output chan<- stage.StreamElement) error

Process appends the preloaded skill instructions to the system prompt.

type StripToolMessagesStage

type StripToolMessagesStage struct {
	stage.BaseStage
}

StripToolMessagesStage removes tool role messages from the stream. This is used in self-play scenarios before calling the self-play provider.

func NewStripToolMessagesStage

func NewStripToolMessagesStage() *StripToolMessagesStage

NewStripToolMessagesStage creates a new strip tool messages stage.

func (*StripToolMessagesStage) Process

func (s *StripToolMessagesStage) Process(
	ctx context.Context,
	input <-chan stage.StreamElement,
	output chan<- stage.StreamElement,
) error

Process filters out elements with tool role messages.

type TTSSentenceAggregatorStage added in v1.5.3

type TTSSentenceAggregatorStage struct {
	stage.BaseStage
	// contains filtered or unexported fields
}

TTSSentenceAggregatorStage turns a streaming provider's token-by-token text deltas into complete sentences for TTS, so the agent speaks in smooth phrases instead of "word … pause … word".

It sits between the AssistantTTSFilterStage (which has already dropped the user transcript) and the TTS stage. As assistant text deltas arrive it accumulates them and emits each complete sentence (ending in . ! or ?) the moment it forms — while the model is still generating the rest. That overlaps generation with speech: audio starts after the first sentence rather than after the whole reply.

The complete assistant Message that the provider emits at the end of a turn duplicates the deltas, so once any delta has been seen the Message is dropped (after flushing the trailing partial sentence). For a NON-streaming provider (no deltas), the Message's content is split into sentences instead.

Control signals pass through: Interrupt clears the buffer (barge-in discards the half-spoken reply); EndOfTurn / EndOfStream flush the trailing buffer.

func NewTTSSentenceAggregatorStage added in v1.5.3

func NewTTSSentenceAggregatorStage() *TTSSentenceAggregatorStage

NewTTSSentenceAggregatorStage creates the sentence aggregator.

func (*TTSSentenceAggregatorStage) Process added in v1.5.3

func (s *TTSSentenceAggregatorStage) Process(
	ctx context.Context,
	input <-chan stage.StreamElement,
	output chan<- stage.StreamElement,
) error

Process implements the Stage interface.

type TurnEvalRunner added in v1.3.2

type TurnEvalRunner interface {
	RunAssertionsAsEvals(
		ctx context.Context,
		assertionConfigs []assertions.AssertionConfig,
		messages []types.Message,
		turnIndex int,
		sessionID string,
		trigger evals.EvalTrigger,
	) []evals.EvalResult
}

TurnEvalRunner is an interface for running assertions as evals during turn execution. EvalOrchestrator in the engine package implements this interface.

type TurnIndexStage

type TurnIndexStage struct {
	stage.BaseStage
	// contains filtered or unexported fields
}

TurnIndexStage computes role-specific turn counters from accumulated messages. It writes the counters into TurnState's ProviderRequestMetadata so downstream stages (e.g. mock provider context) and persisted state.Metadata can read them. Keys written:

  • arena_user_completed_turns: number of completed user messages
  • arena_user_next_turn: completed user messages + 1 (next user turn to generate)
  • arena_assistant_completed_turns: number of completed assistant messages
  • arena_assistant_next_turn: completed assistant messages + 1

func NewTurnIndexStage

func NewTurnIndexStage() *TurnIndexStage

NewTurnIndexStage creates a new turn index stage that does not publish into any TurnState. Useful for tests that only need passthrough behavior.

func NewTurnIndexStageWithTurnState added in v1.4.7

func NewTurnIndexStageWithTurnState(turnState *stage.TurnState) *TurnIndexStage

NewTurnIndexStageWithTurnState creates a turn index stage that publishes the computed counters onto the supplied TurnState's ProviderRequestMetadata.

func (*TurnIndexStage) Process

func (s *TurnIndexStage) Process(ctx context.Context, input <-chan stage.StreamElement, output chan<- stage.StreamElement) error

Process computes role-specific turn counters and forwards elements unchanged.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL