llm

package
v0.74.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 3, 2026 License: BSD-3-Clause, AGPL-3.0 Imports: 6 Imported by: 0

Documentation

Overview

Package llm provides the shared LLM request and response parsing library consumed by proxy middleware. It is runtime agnostic: the same package is used by the native built-in executor now and will be reused by the WASM adapter later.

Index

Constants

View Source 
const ProviderNameBedrock = "bedrock"

ProviderNameBedrock is the stable label for the AWS Bedrock parser, used as the llm.provider metadata value and the cost-meter formula selector.

Variables

View Source 
var (
	// ErrUnknownProvider indicates no parser claimed the request path.
	ErrUnknownProvider = errors.New("llmobs: unknown provider")

	// ErrUnsupportedModel indicates the response parsed successfully but the
	// model is absent from the pricing table. Token counts are still valid.
	ErrUnsupportedModel = errors.New("llmobs: unsupported model")

	// ErrNotLLMResponse indicates the response is not a JSON success body
	// that a non-streaming parser can consume (non-200 or wrong content type).
	ErrNotLLMResponse = errors.New("llmobs: not an LLM response")

	// ErrStreamingUnsupported indicates the caller passed an SSE response to
	// a non-streaming parser. Streaming is handled separately via the SSE
	// scanner.
	ErrStreamingUnsupported = errors.New("llmobs: streaming response requires SSE scanner")

	// ErrMalformedResponse indicates the response body could not be decoded
	// as the provider-specific JSON schema.
	ErrMalformedResponse = errors.New("llmobs: malformed response body")

	// ErrMalformedRequest indicates the request body could not be decoded as
	// the provider-specific JSON schema.
	ErrMalformedRequest = errors.New("llmobs: malformed request body")
)

Sentinel errors returned by parsers and the pricing loader. Callers use errors.Is to branch on a condition without coupling to parser internals.

Functions

This section is empty.

Types

type AnthropicParser 

type AnthropicParser struct{}

AnthropicParser implements the Parser interface for the Anthropic Messages and Completions APIs. Detection is substring-based to tolerate upstream path rewrites.

func (AnthropicParser) DetectFromURL 

func (AnthropicParser) DetectFromURL(path string) bool

DetectFromURL reports whether the given request path looks like an Anthropic API endpoint. The match is case-insensitive and substring-based.

func (AnthropicParser) ExtractCompletion 

func (AnthropicParser) ExtractCompletion(status int, contentType string, body []byte) string

ExtractCompletion returns the assistant text from a non-streaming Anthropic Messages or Completions response. Returns "" when status/content-type indicate the body is not parseable or no text part is present.

func (AnthropicParser) ExtractPrompt 

func (AnthropicParser) ExtractPrompt(body []byte) string

ExtractPrompt returns the user-visible prompt text from an Anthropic request body. Handles the Messages API (system + messages[]) and the legacy /v1/complete prompt string. Returns "" on any decode failure.

func (AnthropicParser) ExtractSessionID 

func (AnthropicParser) ExtractSessionID(body []byte) string

ExtractSessionID is the body-side fallback for Anthropic. Claude Code's authoritative session marker is the X-Claude-Code-Session-Id request header (handled by the request-parser middleware); this only mines the optional metadata.user_id for an embedded "...session_<uuid>" marker. metadata.user_id on its own is a USER identifier, not a session, so the whole value is deliberately NOT used — returning it would mislabel every request from a user as one session. Returns "" when no session marker is present.

func (AnthropicParser) ParseRequest 

func (AnthropicParser) ParseRequest(body []byte) (RequestFacts, error)

ParseRequest extracts the model name and streaming flag from an Anthropic request body. Unknown or missing fields leave the corresponding struct members zero-valued.

func (AnthropicParser) ParseResponse 

func (AnthropicParser) ParseResponse(status int, contentType string, body []byte) (Usage, error)

ParseResponse decodes the non-streaming Anthropic response envelope. Status codes other than 200 are treated as non-LLM responses so the caller can skip cost accounting without aborting the request.

func (AnthropicParser) Provider 

func (AnthropicParser) Provider() Provider

Provider returns ProviderAnthropic.

func (AnthropicParser) ProviderName 

func (AnthropicParser) ProviderName() string

ProviderName returns the stable label used for metrics and metadata.

type BedrockParser 

type BedrockParser struct{}

BedrockParser implements the Parser interface for the AWS Bedrock runtime. Bedrock carries the model in the URL path (/model/{id}/{action}); the request middleware extracts it there, so this parser focuses on the response shapes: the vendor-native InvokeModel body (e.g. Anthropic's snake_case usage) and the unified Converse body (camelCase usage).

func (BedrockParser) DetectFromURL 

func (BedrockParser) DetectFromURL(path string) bool

DetectFromURL reports whether the path is a Bedrock runtime model endpoint.

func (BedrockParser) ExtractCompletion 

func (BedrockParser) ExtractCompletion(status int, contentType string, body []byte) string

ExtractCompletion returns the assistant text from a non-streaming Bedrock response, handling InvokeModel (Anthropic content[].text) and Converse (output.message.content[].text).

func (BedrockParser) ExtractPrompt 

func (BedrockParser) ExtractPrompt(body []byte) string

ExtractPrompt returns the user-visible prompt from a Bedrock request body, handling both the InvokeModel (Anthropic Messages: system + messages[]) and Converse (messages[].content[].text) shapes. Returns "" on decode failure.

func (BedrockParser) ExtractSessionID 

func (BedrockParser) ExtractSessionID([]byte) string

ExtractSessionID has no Bedrock-native marker; session grouping relies on the request headers handled by the middleware. Returns "".

func (BedrockParser) ParseRequest 

func (BedrockParser) ParseRequest([]byte) (RequestFacts, error)

ParseRequest is a no-op for Bedrock: the model lives in the URL path, not the body, and the streaming flag is derived from the path action. The request middleware handles both via parseBedrockPath, so this returns empty facts.

func (BedrockParser) ParseResponse 

func (BedrockParser) ParseResponse(status int, contentType string, body []byte) (Usage, error)

ParseResponse decodes the non-streaming Bedrock response envelope, handling both the InvokeModel and Converse usage shapes. Non-200 / non-JSON bodies are treated as non-LLM responses so the caller skips cost accounting.

func (BedrockParser) Provider 

func (BedrockParser) Provider() Provider

Provider returns ProviderBedrock.

func (BedrockParser) ProviderName 

func (BedrockParser) ProviderName() string

ProviderName returns the stable label used for metrics and metadata.

type Event 

type Event struct {
	Type string
	Data string
}

Event represents a single server-sent event. Type is the dispatch name carried on an "event:" line (empty when the stream uses only "data:" lines). Data is the concatenation of every "data:" line that made up the event, joined by a single newline.

type OpenAIParser 

type OpenAIParser struct{}

OpenAIParser implements the Parser interface for OpenAI-compatible APIs. It recognizes chat.completions, completions, embeddings, and the newer responses endpoint; any proxy path-prefix stripping is tolerated by the substring match in DetectFromURL.

func (OpenAIParser) DetectFromURL 

func (OpenAIParser) DetectFromURL(path string) bool

DetectFromURL reports whether the given request path looks like an OpenAI API endpoint. The match is case-insensitive and substring-based so that a reverse proxy prefix strip or rewrite does not defeat detection.

func (OpenAIParser) ExtractCompletion 

func (OpenAIParser) ExtractCompletion(status int, contentType string, body []byte) string

ExtractCompletion returns the assistant text from a non-streaming OpenAI response. Handles chat.completions (choices[].message.content), legacy completions (choices[].text), and Responses API (output[].content[].text or the convenience output_text field).

func (OpenAIParser) ExtractPrompt 

func (OpenAIParser) ExtractPrompt(body []byte) string

ExtractPrompt returns the user-visible prompt text from an OpenAI request. Handles chat.completions (messages[].content), legacy completions (prompt string), and the Responses API (input as string or content-part array). Returns "" when nothing extractable is found.

func (OpenAIParser) ExtractSessionID 

func (OpenAIParser) ExtractSessionID(body []byte) string

ExtractSessionID reads the OpenAI session marker. Codex (the Responses API client) stamps client_metadata.session_id on every request body; plain chat.completions traffic carries no session id and yields "".

func (OpenAIParser) ParseRequest 

func (OpenAIParser) ParseRequest(body []byte) (RequestFacts, error)

ParseRequest extracts the model name and streaming flag from an OpenAI request body. Unknown or missing fields leave the corresponding struct members zero-valued.

func (OpenAIParser) ParseResponse 

func (OpenAIParser) ParseResponse(status int, contentType string, body []byte) (Usage, error)

ParseResponse decodes the non-streaming OpenAI response envelope. Status codes other than 200 are treated as non-LLM responses so the caller can skip cost accounting without aborting the request.

func (OpenAIParser) Provider 

func (OpenAIParser) Provider() Provider

Provider returns ProviderOpenAI.

func (OpenAIParser) ProviderName 

func (OpenAIParser) ProviderName() string

ProviderName returns the stable label used for metrics and metadata.

type Parser 

type Parser interface {
	Provider() Provider
	ProviderName() string
	DetectFromURL(path string) bool
	ParseRequest(body []byte) (RequestFacts, error)
	ParseResponse(status int, contentType string, body []byte) (Usage, error)
	// ExtractPrompt returns the user-facing prompt text from a request body.
	// Different endpoint shapes (chat.completions, responses, messages) are
	// handled by the per-provider implementation. Returns "" when no prompt
	// can be extracted; never returns an error — extraction is best-effort
	// because callers use the result for observability, not authorization.
	ExtractPrompt(body []byte) string
	// ExtractCompletion returns the assistant-facing completion text from a
	// non-streaming response body. status and contentType match the
	// ParseResponse arguments so implementations can fast-fail uniformly.
	ExtractCompletion(status int, contentType string, body []byte) string
	// ExtractSessionID returns a stable identifier that groups requests of
	// the same conversation / coding session, read from the per-provider
	// location clients populate (e.g. OpenAI Codex's client_metadata.session_id,
	// Claude Code's metadata.user_id). Returns "" when the body carries no
	// recognised session marker; extraction is best-effort and never errors.
	ExtractSessionID(body []byte) string
}

Parser is the per-provider interface implemented in this package. The dispatcher selects a parser by calling DetectFromURL against the incoming request path; ties break by registration order (see Parsers).

func DetectParser 

func DetectParser(path string) (Parser, bool)

DetectParser returns the first parser whose DetectFromURL matches the given request path. ok=false means no parser claimed the path.

func ParserByName 

func ParserByName(id string) (Parser, bool)

ParserByName returns the parser whose ProviderName matches id. Used by callers that already know which provider surface a request will hit (e.g. the agent-network middleware chain configured per synthesised service) so they can skip URL sniffing. ok=false when no parser is registered under that name.

func Parsers 

func Parsers() []Parser

Parsers returns the built-in parser set in a stable order. The order is deterministic so that DetectFromURL ties produce consistent routing.

type Provider 

type Provider int

Provider identifies an LLM API provider. 

const (
	// ProviderUnknown signals that no parser matched the request.
	ProviderUnknown Provider = 0
	// ProviderOpenAI identifies the OpenAI API surface.
	ProviderOpenAI Provider = 1
	// ProviderAnthropic identifies the Anthropic Messages API surface.
	ProviderAnthropic Provider = 2
	// ProviderBedrock identifies the AWS Bedrock runtime surface.
	ProviderBedrock Provider = 3
)

type RequestFacts 

type RequestFacts struct {
	Model  string
	Stream bool
}

RequestFacts captures the subset of the LLM request body that the middleware annotates as metadata (model, streaming flag). Additional fields are added as parsers grow.

type Scanner 

type Scanner struct {
	// contains filtered or unexported fields
}

Scanner reads SSE events from an underlying byte stream. Events are delimited by a blank line ("\n\n"). CRLF line endings are normalized to LF transparently so fixtures captured from live servers can be replayed.

Scanner is not safe for concurrent use.

func NewScanner 

func NewScanner(r io.Reader) *Scanner

NewScanner wraps the given reader. The default underlying buffer size is large enough for typical provider events (~64 KiB); callers needing larger events can wrap the reader in their own bufio.Reader beforehand.

func (*Scanner) Next 

func (s *Scanner) Next() (Event, error)

Next returns the next event. It returns io.EOF after the final event has been consumed. A trailing event that is not terminated by a blank line is still returned before io.EOF so that servers which close the connection without a trailing newline are handled correctly.

type Usage 

type Usage struct {
	InputTokens         int64
	OutputTokens        int64
	TotalTokens         int64
	CachedInputTokens   int64
	CacheCreationTokens int64
}

Usage is the provider-agnostic token accounting emitted to metrics and access logs. Downstream consumers map InputTokens/OutputTokens to the plg.llm.* metadata allowlist entries.

CachedInputTokens carries OpenAI's prompt_tokens_details.cached_tokens (a SUBSET of InputTokens) when the response is from OpenAI, or Anthropic's cache_read_input_tokens (ADDITIVE to InputTokens) when from Anthropic. The cost meter switches formula on KeyLLMProvider so the two shapes are billed correctly without double-counting.

CacheCreationTokens carries Anthropic's cache_creation_input_tokens (ADDITIVE; not present in the OpenAI shape).

Source Files

View all Source files

Directories

Path Synopsis
Package pricing implements the embedded-default + override pricing table shared by middleware that converts LLM token usage into a USD cost estimate.
 Package pricing implements the embedded-default + override pricing table shared by middleware that converts LLM token usage into a USD cost estimate.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.