reasoninge2e

package
v0.1.0-rc.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 1, 2026 License: Apache-2.0 Imports: 18 Imported by: 0

Documentation

Overview

Package reasoninge2e provides a deterministic client-transcript model and backend-request oracle for reasoning-preservation full HTTP E2E phases.

Plans are precomputed from an explicit seed and retention policy. The package never uses package-global RNG or time.Now, returns defensive copies, and keeps oracle errors content-safe (no reasoning text, signatures, or opaque payloads).

GenerateTranscriptPlan builds immutable matrix TranscriptPlans (seed+mode+turns) with independent backend/client RNG streams and forced coverage categories.

ClientEmulator records actual proxy ChatResponse observations against Plan.Observed, then materializes the next Chat request from recorded visible/tool structure plus policy-specific submitted reasoning. Observed and submitted histories stay independent.

Responses stateful exact-reasoning harness helpers live alongside Chat (see responses.go); wire scripts/oracles remain in refbackend/refclient openairesponses packages.

AssistantTurn.Streaming is plan/client metadata; Check / BackendTurnObservation do not validate streaming wire shape (HTTP drivers assert Content-Type / SSE framing).

CheckPrefixRetention models FIFO store bounds for ModeDropped restoration; Check and CheckPrefix remain eviction-blind. CheckPrefixRetention requires maxArtifactTurns > 0.

Index

Constants

View Source
const (
	DialectOpenAIChatTextV1            = "openai.chat.reasoning_text.v1"
	DialectAnthropicThinkingV1         = "anthropic.thinking.v1"
	DialectAnthropicRedactedThinkingV1 = "anthropic.redacted_thinking.v1"
)
View Source
const (
	EnvSoakGate    = "LIP_REASONING_E2E_SOAK"
	EnvSoakSeeds   = "LIP_REASONING_E2E_SEEDS"
	EnvSoakTurns   = "LIP_REASONING_E2E_TURNS"
	EnvSoakWorkers = "LIP_REASONING_E2E_WORKERS"
	EnvSoakMode    = "LIP_REASONING_E2E_MODE"
	EnvSoakSeed    = "LIP_REASONING_E2E_SEED"
)

Soak environment contract (opt-in; never a default/PR gate).

View Source
const (
	// DefaultSoakSeeds is the default seed count (1000) for a full soak.
	DefaultSoakSeeds = 1000
	// DefaultSoakTurns is the default HTTP turns per seed (100).
	DefaultSoakTurns = 100
	// DefaultSoakWorkers is a conservative fixed-pool default.
	DefaultSoakWorkers = 4
)

Variables

View Source
var ErrSoakDisabled = errors.New("reasoninge2e soak: disabled (set LIP_REASONING_E2E_SOAK=1)")

ErrSoakDisabled is returned by helpers that require an enabled soak gate.

Functions

func AssistantTurnToChatMessage

func AssistantTurnToChatMessage(turn AssistantTurn) map[string]any

AssistantTurnToChatMessage converts a planned/submitted assistant turn to wire JSON.

func Check

func Check(plan Plan, obs BackendRequestObservation) error

Check compares a backend-bound request observation against the precomputed plan. Errors include seed, turn id, and mode, and describe structural mismatches only. Streaming on PlannedTurn / AssistantTurn is plan metadata only; this oracle does not validate stream vs non-stream wire shape on BackendTurnObservation.

func CheckPrefix

func CheckPrefix(plan Plan, obs BackendRequestObservation) error

CheckPrefix validates a prefix of the plan against a backend observation. Used by multi-turn HTTP drivers where each request carries only history to date.

func CheckPrefixRetention

func CheckPrefixRetention(plan Plan, obs BackendRequestObservation, maxArtifactTurns int) error

CheckPrefixRetention validates a plan prefix against a backend observation while modeling FIFO retention of artifact-producing turns (non-empty observed reasoning).

maxArtifactTurns is the store bound (newest N artifact turns retained) and must be > 0. Use CheckPrefix for unbounded/eviction-blind validation.

ModeDropped expects restoration only while retained; after eviction it expects absence. ModePreserved / ModeConflict / ModeNone keep CheckPrefix semantics. The immutable plan ExpectedBackend fields are never mutated.

func CheckResponsesHistoryIDs

func CheckResponsesHistoryIDs(got, want []string) error

func ClientRetainSequence

func ClientRetainSequence(seed uint64, n int) []bool

ClientRetainSequence returns the independent client preserve bits for seed (true=preserve). Exported for independence proofs; callers must not treat this as payload-bearing.

func FormatFail

func FormatFail(plan Plan, turnID string, mode RetentionMode, field, detail string) string

FormatFail wraps an oracle/driver error with content-safe seed/mode/turn context.

func FormatMatrixFail

func FormatMatrixFail(plan TranscriptPlan, turnIndex int, reasonCode string) string

FormatMatrixFail builds a content-safe matrix failure line with replay command.

func FormatRetentionDiag

func FormatRetentionDiag(err error) string

FormatRetentionDiag extracts content-safe retention fields from an oracle error for matrix/soak fail lines. Returns "" when err is nil or carries no retention diagnostics.

func FormatSoakFail

func FormatSoakFail(plan TranscriptPlan, turnIndex int, reasonCode string) string

FormatSoakFail builds a content-safe soak failure line with a single replay command.

func MatrixReplayCommand

func MatrixReplayCommand(mode MatrixMode, seed uint64) string

MatrixReplayCommand returns a single content-safe seed-replay command.

func RunSoakWorkerPoolWithStats

func RunSoakWorkerPoolWithStats(workers int, jobs []SoakPoolJob, run func(SoakPoolJob) SoakPoolResult) ([]SoakPoolResult, SoakPoolRunStats)

RunSoakWorkerPoolWithStats is RunSoakWorkerPool plus worker/job concurrency stats.

func SoakReplayCommand

func SoakReplayCommand(mode MatrixMode, seed uint64) string

SoakReplayCommand returns one content-safe command to reproduce a soak seed.

func ToolResultMessage

func ToolResultMessage(tool ToolExchange) map[string]any

ToolResultMessage builds a role=tool wire message.

func UserMessage

func UserMessage(text string) map[string]any

UserMessage builds a role=user wire message.

Types

type AssistantTurn

type AssistantTurn struct {
	ID          string
	VisibleText string
	Reasoning   []ReasoningBlock
	Tool        *ToolExchange
	Streaming   bool
}

AssistantTurn is one assistant turn in an observed or submitted transcript. Streaming is plan/client metadata (how the turn was produced); Check does not read it from BackendTurnObservation.

func ParseChatJSONAssistant

func ParseChatJSONAssistant(body []byte) (AssistantTurn, error)

ParseChatJSONAssistant extracts assistant fields from a non-stream chat.completion body.

func ParseChatSSEAssistant

func ParseChatSSEAssistant(body []byte) (AssistantTurn, error)

ParseChatSSEAssistant aggregates assistant fields from a chat.completion.chunk SSE body.

type BackendRequestObservation

type BackendRequestObservation struct {
	AssistantTurns []BackendTurnObservation
}

BackendRequestObservation is the oracle input for one backend-bound request.

func ObserveChatBackendRequest

func ObserveChatBackendRequest(body []byte, turnIDs []string) (BackendRequestObservation, error)

ObserveChatBackendRequest parses a backend-bound chat/completions body into oracle input. turnIDs must align with assistant history order (may be shorter than plan).

type BackendTurnObservation

type BackendTurnObservation struct {
	TurnID      string
	VisibleText string
	Reasoning   []ReasoningBlock
	Tool        *ToolExchange
}

BackendTurnObservation is one historical assistant turn as seen on a backend-bound request. It has no Streaming field: stream vs non-stream wire shape is out of oracle scope.

type ChatCompletionRequest

type ChatCompletionRequest struct {
	Model    string           `json:"model"`
	Stream   bool             `json:"stream,omitempty"`
	Messages []map[string]any `json:"messages"`
	Tools    []map[string]any `json:"tools,omitempty"`
}

ChatCompletionRequest is a minimal wire request body.

type ChatResponse

type ChatResponse struct {
	VisibleText string
	Reasoning   []ReasoningBlock
	Tool        *ToolExchange
	Streaming   bool
}

ChatResponse is one proxy assistant observation recorded by the client emulator. Tool.Result is optional on the wire response; materialization uses the plan tool result.

func ChatResponseFromTurn

func ChatResponseFromTurn(resp ChatTurnResponse) ChatResponse

ChatResponseFromTurn maps a wire client response into a ClientEmulator ChatResponse.

type ChatSessionCarriers

type ChatSessionCarriers struct {
	SessionID   string
	ResumeToken string
}

ChatSessionCarriers holds authoritative session resume material from proxy responses.

type ChatTurnResponse

type ChatTurnResponse struct {
	Status      int
	RawBody     []byte
	ContentType string
	Stream      bool
	VisibleText string
	Reasoning   []ReasoningBlock
	Tool        *ToolExchange
	Carriers    ChatSessionCarriers
}

ChatTurnResponse is the parsed proxy response for one chat turn.

type ChatWireClient

type ChatWireClient struct {
	BaseURL    string
	APIKey     string
	HTTPClient *http.Client
	Model      string
	// Route, when non-empty, sets X-LIP-Route on each request (overrides proxy default_route).
	Route    string
	Carriers ChatSessionCarriers
}

ChatWireClient is a stateful raw-wire OpenAI Chat Completions client for E2E drivers.

func (*ChatWireClient) PostChatCompletion

func (c *ChatWireClient) PostChatCompletion(ctx context.Context, stream bool, messages []map[string]any, tools []map[string]any) (ChatTurnResponse, error)

PostChatCompletion sends one raw chat/completions request and updates carriers.

type ClientEmulator

type ClientEmulator struct {
	// contains filtered or unexported fields
}

ClientEmulator (Conversation) owns an immutable Plan plus recorded actual proxy outputs. It keeps observed and submitted histories independently, returns defensive copies, and never calls testing.T or logs payloads.

func NewClientEmulator

func NewClientEmulator(plan Plan) *ClientEmulator

NewClientEmulator returns a conversation emulator bound to plan.

func (*ClientEmulator) MaterializeChatRequest

func (e *ClientEmulator) MaterializeChatRequest(nextUserPrompt string) ([]map[string]any, error)

MaterializeChatRequest builds the next Chat Completions messages list. History assistant visible/tool structure comes from recorded actual responses; reasoning follows the plan retention policy (submitted). Tool results and prior user prompts are included. It is an error to materialize again while a prior materialize is still awaiting Record (unrecorded turn).

func (*ClientEmulator) ObservedHistory

func (e *ClientEmulator) ObservedHistory() []AssistantTurn

ObservedHistory returns defensive copies of recorded actual assistant observations.

func (*ClientEmulator) Plan

func (e *ClientEmulator) Plan() Plan

Plan returns the immutable bound plan (defensive turn copies via Plan methods).

func (*ClientEmulator) Record

func (e *ClientEmulator) Record(resp ChatResponse) error

Record validates resp against the next Plan.Observed turn (structural, content-safe), then appends independent observed and submitted history entries. Visible/tool on submitted come from the actual recorded response; reasoning follows plan policy.

func (*ClientEmulator) SubmittedHistory

func (e *ClientEmulator) SubmittedHistory() []AssistantTurn

SubmittedHistory returns defensive copies of policy-materialized submitted turns.

type DecisionRecord

type DecisionRecord struct {
	Index          int
	TurnID         string
	BackendKind    string // reason | no_reason | tool
	ClientDecision string // preserve | drop | none
	Streaming      bool
	HasReasoning   bool
	HasTool        bool
	ReasonCode     string // structural label only; empty unless set by callers for failures
}

DecisionRecord is one compact structural decision for a generated turn.

type FailTrace

type FailTrace struct {
	Seed   uint64
	Policy string
	Mode   string
	TurnID string
	Field  string
	Detail string
}

FailTrace is a content-safe structural failure description for E2E drivers. It must never carry reasoning text, signatures, opaque payloads, or anchors.

func (FailTrace) String

func (f FailTrace) String() string

type MatrixCase

type MatrixCase struct {
	Mode MatrixMode
	Seed uint64
}

MatrixCase is one default-matrix (mode, seed) pair.

func DefaultMatrixCases

func DefaultMatrixCases() []MatrixCase

DefaultMatrixCases returns the exact 64-seed split: 16 drop-all, 16 always-reason, 32 combined. Always-reason and combined seeds are chosen so the independent client RNG stream naturally includes both preserve and drop (client bits are never mutated).

func SoakCases

func SoakCases(cfg SoakConfig) []MatrixCase

SoakCases returns the mode/seed allocation for cfg. Defaults preserve 25%/25%/50% (250/250/500 of 1000). Replay returns one case.

type MatrixMode

type MatrixMode string

MatrixMode selects a precomputed random-matrix scenario family.

const (
	// MatrixModeRandomBackendDropAll randomizes backend reason/no-reason/tool; client drops all.
	MatrixModeRandomBackendDropAll MatrixMode = "random_backend_drop_all"
	// MatrixModeAlwaysReasonRandomClient always emits reasoning; client preserve/drop is randomized.
	MatrixModeAlwaysReasonRandomClient MatrixMode = "always_reason_random_client"
	// MatrixModeCombined randomizes backend and client retention independently.
	MatrixModeCombined MatrixMode = "combined"
)

type Plan

type Plan struct {
	Seed   uint64
	Policy RetentionPolicy
	// contains filtered or unexported fields
}

Plan is a precomputed deterministic transcript plan.

func BuildPlan

func BuildPlan(cfg PlanConfig) (Plan, error)

BuildPlan precomputes a deterministic transcript plan from an explicit seed and policy.

func (Plan) ObservedTranscript

func (p Plan) ObservedTranscript() []AssistantTurn

ObservedTranscript returns a defensive copy of the immutable observed assistant turns.

func (Plan) SubmittedTranscript

func (p Plan) SubmittedTranscript() []AssistantTurn

SubmittedTranscript returns a defensive copy of the materialized client-submitted turns.

func (Plan) Turns

func (p Plan) Turns() []PlannedTurn

Turns returns defensive copies of planned turns.

type PlanConfig

type PlanConfig struct {
	Seed   uint64
	Policy RetentionPolicy
	Turns  []TurnSpec
}

PlanConfig is the explicit deterministic plan input.

type PlannedTurn

type PlannedTurn struct {
	ID              string
	Mode            RetentionMode
	Observed        AssistantTurn
	Submitted       AssistantTurn
	ExpectedBackend AssistantTurn
}

PlannedTurn is the precomputed expectation for one assistant turn.

type ReasoningBlock

type ReasoningBlock struct {
	Dialect   string
	Text      string
	Signature string
	Opaque    []byte
}

ReasoningBlock is one ordered reasoning payload on an assistant turn.

type ResponsesMatrixCase

type ResponsesMatrixCase struct {
	Seed     uint64
	Variant  ResponsesPresenceVariant
	StreamFE bool
	Trace    string
}

ResponsesMatrixCase is one reproducible Responses-inclusive smoke cell.

func DefaultResponsesSmokeCases

func DefaultResponsesSmokeCases() []ResponsesMatrixCase

DefaultResponsesSmokeCases returns a moderate fixed-seed set (8 cells) for local smoke.

func ResponsesSmokeCases

func ResponsesSmokeCases(seedBase uint64, n int) []ResponsesMatrixCase

ResponsesSmokeCases builds n deterministic Responses presence/stream cells from seedBase.

type ResponsesPresenceVariant

type ResponsesPresenceVariant string

ResponsesPresenceVariant is a deterministic exact-item presence case for seeded smokes.

const (
	ResponsesPresenceEncryptedAbsent ResponsesPresenceVariant = "enc_absent"
	ResponsesPresenceEncryptedNull   ResponsesPresenceVariant = "enc_null"
	ResponsesPresenceEncryptedValue  ResponsesPresenceVariant = "enc_value"
	ResponsesPresenceWithContent     ResponsesPresenceVariant = "with_content"
)

type RetentionMode

type RetentionMode string

RetentionMode is the planned fate of one assistant turn's reasoning.

const (
	// ModeNone means the observed turn had no reasoning; nothing may be inserted later.
	ModeNone RetentionMode = "none"
	// ModePreserved means the client still carries the observed reasoning.
	ModePreserved RetentionMode = "preserved"
	// ModeDropped means the client omitted observed reasoning (restore candidate).
	ModeDropped RetentionMode = "dropped"
	// ModeConflict means the client submitted conflicting reasoning that must stay untouched.
	ModeConflict RetentionMode = "conflict"
)

type RetentionPolicy

type RetentionPolicy int

RetentionPolicy controls how the client materializes previously observed assistant reasoning.

const (
	// PreserveAllReasoning keeps every observed reasoning block in the submitted transcript.
	PreserveAllReasoning RetentionPolicy = iota
	// DropAllReasoning strips all observed reasoning from the submitted transcript.
	DropAllReasoning
	// SeededPerTurnRetention keeps or drops each turn's reasoning using a seeded PRNG.
	SeededPerTurnRetention
	// ConflictReasoning submits alternate reasoning that differs from the observed blocks.
	ConflictReasoning
)

func (RetentionPolicy) String

func (p RetentionPolicy) String() string

type ScriptedBackendTurn

type ScriptedBackendTurn struct {
	VisibleText   string
	ReasoningText string
	ToolID        string
	ToolName      string
	ToolArgs      string
}

ScriptedBackendTurn is a content-bearing scripted assistant response for refbackends. HTTP drivers map this to emulator ScriptedTurn values; traces must not print these fields.

type SoakConfig

type SoakConfig struct {
	Enabled bool
	Seeds   int
	Turns   int
	Workers int
	Replay  *SoakReplay
}

SoakConfig is the validated soak driver configuration.

func LoadSoakConfigFromEnv

func LoadSoakConfigFromEnv() (SoakConfig, error)

LoadSoakConfigFromEnv parses soak settings from the process environment.

func ParseSoakConfig

func ParseSoakConfig(getenv func(string) string) (SoakConfig, error)

ParseSoakConfig parses soak settings from getenv. When the soak gate is unset, overrides are ignored and Enabled is false (no error).

type SoakPoolJob

type SoakPoolJob struct {
	Index int
	Case  MatrixCase
}

SoakPoolJob is one unit of work for the fixed soak worker pool.

type SoakPoolResult

type SoakPoolResult struct {
	Index     int
	Case      MatrixCase
	HTTPTurns int
	Err       error
}

SoakPoolResult is the outcome of one soak job (content-safe Err only).

func RunSoakWorkerPool

func RunSoakWorkerPool(workers int, jobs []SoakPoolJob, run func(SoakPoolJob) SoakPoolResult) []SoakPoolResult

RunSoakWorkerPool runs jobs with exactly workers long-lived goroutines. It does not spawn one goroutine per job. run must not call testing.T methods.

type SoakPoolRunStats

type SoakPoolRunStats struct {
	WorkersStarted int64
	JobsStarted    int64
	MaxActive      int64
}

SoakPoolRunStats captures fixed-pool execution metrics (worker spawn vs job count).

type SoakReplay

type SoakReplay struct {
	Mode MatrixMode
	Seed uint64
}

SoakReplay identifies a single mode/seed pair for reproduction.

type ToolExchange

type ToolExchange struct {
	ID        string
	Name      string
	Arguments string
	Result    string
}

ToolExchange is an optional tool call plus result attached to an assistant turn.

type TranscriptPlan

type TranscriptPlan struct {
	// contains filtered or unexported fields
}

TranscriptPlan is an immutable precomputed matrix scenario.

func GenerateTranscriptPlan

func GenerateTranscriptPlan(mode MatrixMode, seed uint64, turnCount int) (TranscriptPlan, error)

GenerateTranscriptPlan precomputes a deterministic scenario from seed+mode+turnCount. It uses independent RNG streams for backend and client choices (no package-global RNG).

func (TranscriptPlan) Decisions

func (p TranscriptPlan) Decisions() []DecisionRecord

Decisions returns defensive copies of structural decision records.

func (TranscriptPlan) Mode

func (p TranscriptPlan) Mode() MatrixMode

Mode returns the matrix mode.

func (TranscriptPlan) Plan

func (p TranscriptPlan) Plan() Plan

Plan returns the immutable reasoninge2e Plan (defensive turn copies via Plan methods).

func (TranscriptPlan) ScriptedTurns

func (p TranscriptPlan) ScriptedTurns() []ScriptedBackendTurn

ScriptedTurns returns defensive copies of scripted backend turns.

func (TranscriptPlan) Seed

func (p TranscriptPlan) Seed() uint64

Seed returns the scenario seed.

func (TranscriptPlan) StructuralTrace

func (p TranscriptPlan) StructuralTrace() string

StructuralTrace returns a compact content-safe decision trace.

func (TranscriptPlan) TurnCount

func (p TranscriptPlan) TurnCount() int

TurnCount returns the configured turn count.

type TurnSpec

type TurnSpec struct {
	VisibleText       string
	Reasoning         []ReasoningBlock
	ConflictReasoning []ReasoningBlock
	Tool              *ToolExchange
	Streaming         bool
	// ClientMode, when non-empty, overrides policy-derived retention for this turn.
	ClientMode RetentionMode
}

TurnSpec is the explicit seed input for one observed assistant turn.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL