requestvalidity

package
v0.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 5, 2026 License: MIT Imports: 9 Imported by: 0

Documentation

Overview

Package requestvalidity checks the requests PiG sends to model providers against each provider's documented message contract.

The checker is inert in a production binary. It runs in every Go test binary and in a binary built with the requestvalidity tag. A violation is a bug in the code that built the request, whatever the provider would answer: it ends the process with a stack trace (a panic on a goroutine nothing can recover), so the test or scenario that sent the request fails even when an agent loop wraps the provider call. PIG_REQUEST_VALIDITY_REPORT=<file> appends one line per violation to the file and continues, to survey a whole suite.

Index

Constants

This section is empty.

Variables

View Source
var (
	// Anthropic Messages API: "tool_use ids were found without tool_result blocks immediately after", "unexpected tool_use_id found in tool_result blocks. Each tool_result block must have a corresponding tool_use block in the previous message", "tool_use ids must be unique"; tool_result blocks must come first in the user message; text content blocks must be non-empty; the id matches ^[a-zA-Z0-9_-]+$ (max 64); a final assistant message is a prefill that current models reject. A "system" message inside messages is the mid-conversation system message of the mid-conversation-output-config beta (anthropic-messages.ts MID_CONVERSATION_OUTPUT_CONFIG_BETA).
	Anthropic = Contract{
		API: "anthropic-messages", Cite: "https://docs.claude.com/en/api/messages (tool use, tool_result ordering, prefill)",
		Roles: []string{"system", "user", "assistant"}, ResultsInUserTurn: true, UniqueCallIDs: true,
		IDPattern: lazyregexp.New(`^[a-zA-Z0-9_-]{1,64}$`), NonEmpty: true, LastRoles: []string{"user"},
	}
	// Bedrock Converse: toolResult blocks for every toolUse of the previous assistant message, in the next user message ("Expected toolResult blocks at messages.N.content for Ids"); roles alternate and start with user; toolUseId matches [a-zA-Z0-9_-]+ (max 64); content blocks are non-empty.
	Bedrock = Contract{
		API: "bedrock-converse-stream", Cite: "https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_Converse.html (ToolUseBlock, ToolResultBlock, Message)",
		Roles: []string{"user", "assistant"}, ResultsInUserTurn: true, UniqueCallIDs: true,
		IDPattern: lazyregexp.New(`^[a-zA-Z0-9_-]{1,64}$`), Alternate: true, FirstUser: true, NonEmpty: true, LastRoles: []string{"user"},
	}
	// OpenAI Chat Completions: the API answers "An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'" and rejects a tool message that does not answer the tool_calls of the message before it.
	OpenAICompletions = Contract{
		API: "openai-completions", Cite: "https://platform.openai.com/docs/api-reference/chat/create (messages: assistant tool_calls, tool tool_call_id)",
		Roles: []string{"system", "developer", "user", "assistant", "tool"},
	}
	// Mistral chat: tool messages answer the tool_calls of the previous assistant message; the API answers a request whose last role is assistant with "Expected last role User or Tool (or Assistant with prefix True) for serving but got assistant". A user message after tool messages is not a rule: Pi's own Mistral end-to-end tests send it; the tool call id is exactly 9 alphanumeric characters.
	Mistral = Contract{
		API: "mistral-conversations", Cite: "https://docs.mistral.ai/capabilities/function_calling/ and the API's 400 text",
		Roles: []string{"system", "user", "assistant", "tool"}, LastRoles: []string{"user", "tool"},
		IDPattern: lazyregexp.New(`^[a-zA-Z0-9]{9}$`),
	}
	// OpenAI Responses (also Azure and Codex): every function_call needs a function_call_output with the same call_id ("No tool output found for function call"), and every output needs an earlier call ("No tool call found for function call output with call_id"). That an output follows its call's assistant items with nothing between them is Pi's transformMessages invariant (transform-messages.ts closePendingToolCalls), not a documented API error.
	OpenAIResponses = Contract{
		API: "openai-responses", Cite: "https://platform.openai.com/docs/api-reference/responses/create (input: function_call, function_call_output)",
		Roles: []string{"system", "developer", "user", "assistant", "tool"},
	}
	// Gemini generateContent: "Please ensure that the number of function response parts is equal to the number of function call parts of the function call turn", and "function call turn comes immediately after a user turn or after a function response turn"; parts must not be empty.
	Google = Contract{
		API: "google-generative-ai", Cite: "https://ai.google.dev/gemini-api/docs/function-calling (function response turn) and the API's 400 messages",
		Roles: []string{"user", "assistant"}, ResultsInUserTurn: true, CallsFollowUser: true, NonEmpty: true,
	}
	// Transcript is Pi's own message contract after transformMessages: results follow the assistant message that called them, with nothing between, and no call goes unanswered. The pi-messages backend and the faux providers receive this shape.
	Transcript = Contract{
		API: "transcript", Cite: ".upstream/current/packages/ai/src/api/transform-messages.ts (closePendingToolCalls)",
		Roles: []string{"system", "user", "assistant", "tool"},
	}
)

The contracts below state what each provider documents and answers with HTTP 400. Pi 1.1.0 packages/ai/src/api/transform-messages.ts closes unanswered calls and holds system messages, so a request built from a transformed transcript meets the tool-flow rules; a rule here that Pi's own converter does not meet is a finding against Pi.

Functions

func Allow

func Allow(ctx context.Context) context.Context

Allow returns a context whose requests are not checked. A test that sends an invalid request on purpose, to observe how a provider answers it, uses it.

func Enabled

func Enabled() bool

Enabled reports whether provider requests are checked: in a test binary or a requestvalidity-tagged build.

func Enforce

func Enforce(ctx context.Context, api string, violations []Violation)

Enforce fails the request that built an invalid message list.

Types

type Collector

type Collector struct {
	// contains filtered or unexported fields
}

Collector receives the violations of the requests sent under its context, instead of the panic Enforce raises. A fuzz or exploration harness that needs the verdict of every request uses it.

func Collect

func Collect(ctx context.Context) (context.Context, *Collector)

Collect returns a context whose invalid requests are recorded in the collector.

func CollectAll

func CollectAll(t testing.TB) *Collector

CollectAll records the violations of every request the process sends until the test ends, including requests whose context does not carry a collector: a Session's agent loop runs a provider call under its own context. Only one test at a time may collect, and tests that run in parallel with it see their violations recorded too.

func (*Collector) Violations

func (c *Collector) Violations() []string

Violations returns the recorded violations as "api rule: detail" lines.

type Contract

type Contract struct {
	// API is the PiG API name.
	API string
	// Cite names the documentation the rules come from.
	Cite string
	// Roles are the roles the request may contain.
	Roles []string
	// ResultsInUserTurn means the results of one assistant turn are blocks of the single user turn that follows it (Anthropic, Bedrock, Google). Otherwise each result is its own "tool" turn.
	ResultsInUserTurn bool
	// UniqueCallIDs rejects a call id that appears in two assistant turns.
	UniqueCallIDs bool
	// IDPattern is the format a call id must match.
	IDPattern *lazyregexp.Regexp
	// Alternate requires user and assistant turns to alternate.
	Alternate bool
	// FirstUser requires the first non-system turn to be a user turn.
	FirstUser bool
	// LastRoles are the roles the last turn may have. Empty allows any.
	LastRoles []string
	// CallsFollowUser requires an assistant turn with tool calls to follow a user turn.
	CallsFollowUser bool
	// NonEmpty rejects a user or assistant turn with no content.
	NonEmpty bool
}

Contract is the part of a provider's message contract that PiG checks.

func ForAPI

func ForAPI(api string) (Contract, bool)

ForAPI returns the contract of a PiG API.

func (Contract) Check

func (c Contract) Check(turns []Turn) []Violation

Check returns every rule of the contract that turns breaks.

type Turn

type Turn struct {
	// Role is "system", "user", "assistant" or "tool".
	Role string
	// Calls are the tool call ids this turn announces.
	Calls []string
	// Results are the tool call ids this turn answers.
	Results []string
	// Empty is a turn with no content, which some providers reject.
	Empty bool
	// ContentBeforeResult is a turn that carries a tool result after other content.
	ContentBeforeResult bool
}

Turn is one entry of a request's message list, reduced to what a provider's tool contract constrains.

type Violation

type Violation struct {
	// Rule names the broken rule.
	Rule string
	// Detail states which messages break it.
	Detail string
}

Violation is one broken rule of a provider's message contract.

func CheckBody

func CheckBody(api string, body []byte) []Violation

CheckBody parses a provider's JSON request body and returns the contract violations of its message list. A body that is not the API's shape is itself a violation.

func (Violation) String

func (v Violation) String() string

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL