Documentation
¶
Overview ¶
Package ai renders and relays AI model streams over HTTP.
It is transport-aware and provider-agnostic. It understands the shapes that streamed generation takes - text deltas, reasoning traces, tool-call fragments, usage trailers, an error that arrives after the response has already begun - and it ships no vendor SDK, no HTTP client for any AI service, and no vendor request or response schema. What travels over the wire is your business; the shape of the wire is this package's.
The core is a small neutral vocabulary, Part, and a pluggable Encoder that renders it in a wire dialect. One handler serves both a streaming and a non-streaming request, because that is what every OpenAI-shaped API has to do and nothing in Go offers it.
It deliberately does not include an agent loop, a conversation store, a prompt template format, a token counter or a cost table. Those vary too much between applications to belong in a framework, and every framework that shipped one has spent the years since adding escape hatches to it.
Index ¶
- Constants
- Variables
- func Relay(r *Response, resp *http.Response, dec Decoder, opts ...RelayOptions) error
- type Class
- type ClassifyOptions
- type Decoder
- type DecoderFunc
- type Disconnect
- type Encoder
- type Kind
- type Message
- type Options
- type Part
- func (p Part) Args() jsontext.Value
- func (p Part) ArgsAs[T any]() (T, error)
- func (p Part) Delta() string
- func (p Part) Err() error
- func (p Part) ID() string
- func (p Part) IsZero() bool
- func (p Part) Kind() Kind
- func (p Part) MarshalJSON() ([]byte, error)
- func (p Part) MarshalJSONTo(enc *jsontext.Encoder) error
- func (p Part) Name() string
- func (p Part) Raw() jsontext.Value
- func (p Part) Result() jsontext.Value
- func (p Part) ResultAs[T any]() (T, error)
- func (p Part) String() string
- func (p Part) Usage() Usage
- type RelayOptions
- type Response
- type Result
- type Sink
- type TagSplitter
- type UpstreamError
- func (e *UpstreamError) Error() string
- func (e *UpstreamError) ErrorCode() errors.ErrorCodeName
- func (e *UpstreamError) HTTPStatus() int
- func (e *UpstreamError) Report(ctx *context.Context) *errors.Error
- func (e *UpstreamError) Retryable() bool
- func (e *UpstreamError) Timeout() bool
- func (e *UpstreamError) Unwrap() error
- type Usage
Constants ¶
const DefaultMaxErrorBody = 64 << 10
DefaultMaxErrorBody caps how much of a failed response Classify keeps.
Variables ¶
var ErrDropPart = errors.New("iris: ai: drop part")
ErrDropPart tells Open to discard the current part and keep going. Return it from Options.OnPart.
var ErrNonEventStream = errors.New("iris: ai: upstream response is not an event stream")
ErrNonEventStream reports that an upstream answered 200 with a body that is not an event stream.
var ErrNotFramed = errors.New("iris: ai: sink is not an event stream")
ErrNotFramed reports that a Sink does not carry SSE frames, which is the case for a line-oriented dialect such as NDJSON.
var ErrNotRaw = errors.New("iris: ai: sink takes frames, not raw bytes")
ErrNotRaw reports that a Sink does not take raw bytes, which is the case for an SSE dialect, where everything must be a frame.
Functions ¶
func Relay ¶
Relay reads an upstream event stream, decodes it and forwards the parts to r.
It closes resp.Body. It classifies the response first, so a non-2xx upstream comes back as a typed *UpstreamError rather than as an empty stream, and it checks the content type, so a gateway that answers 200 with a JSON error does not silently become a blank answer. A failed read of the upstream body - a reset, a timeout, a stream cut mid-event or closed without the sentinel RelayOptions.Read.Terminators promised - comes back through Classify as a transient *UpstreamError wrapping the cause (io.ErrUnexpectedEOF for the last two), so the client gets a BAD_GATEWAY error frame rather than a terminator claiming the answer was complete; a client that hangs up mid-read comes back as ClassCanceled. Decoder errors are returned as they are: they are the application's own.
resp, err := httpClient.Do(req)
if ue := ai.Classify(resp, err); ue != nil {
return res.Close(ue)
}
return res.Close(ai.Relay(res, resp, decoder, ai.RelayOptions{
Read: sse.ReadOptions{Terminators: []string{"[DONE]"}},
}))
Types ¶
type Class ¶
type Class uint8
Class is what a caller actually needs to decide from an upstream failure: whether to retry, whether to try a different credential, and whether to give up.
It exists because the alternative is what every AI gateway ends up with - a regular expression over err.Error(), because the status code was flattened into a string three packages ago and there is nowhere else left to read it.
const ( // ClassNone means the response was fine. ClassNone Class = iota // ClassFatal is a request-shaped failure: the request is wrong and would be // just as wrong on another credential or another model. Retrying spends // money to get the same answer. ClassFatal // ClassTransient is a failure that may not recur: a 5xx, a reset, a dial // error. Retrying is reasonable. ClassTransient // ClassRateLimited is a 429. It is deliberately its own class rather than a // kind of exhaustion: it is a routing signal, not a health signal, and a // credential pool that counts it as a fault will quarantine a perfectly // healthy key for being popular. ClassRateLimited // ClassExhausted means this credential or account cannot serve the request // at all: it is unauthorised, out of quota, or suspended. A different // credential might work; this one will not, for a while. ClassExhausted // ClassCanceled means the caller went away. It is not a failure of the // upstream, and a second attempt would spend tokens on a response nobody is // waiting for. ClassCanceled )
type ClassifyOptions ¶
type ClassifyOptions struct {
// MaxBody caps how much of the response body is read and kept. Default
// DefaultMaxErrorBody.
MaxBody int64
}
ClassifyOptions tune Classify.
type Decoder ¶
Decoder turns one upstream event into zero or more neutral parts.
This is where a provider's dialect is understood, and it is the caller's code rather than the framework's: the framework has no business knowing what a particular vendor calls a delta this month.
type DecoderFunc ¶
DecoderFunc adapts a function to Decoder.
type Disconnect ¶
type Disconnect uint8
Disconnect decides what a vanished client does to the work in progress.
const ( // DisconnectCancel ends the turn as soon as the client goes away. It is the // default because an abandoned generation burns tokens nobody will read. DisconnectCancel Disconnect = iota // DisconnectDetach keeps consuming the upstream after the client has gone, // so OnFinish still records what was actually spent. // // It does not mean the handler returns and work continues in the background. // The handler still runs to completion; only the client is gone. DisconnectDetach )
type Encoder ¶
type Encoder interface {
// ContentType is the response media type. Empty means text/event-stream and
// selects the SSE transport.
ContentType() string
// Header applies any dialect headers before anything is written.
Header(h http.Header)
// Open is called once, before the first part.
Open(s Sink) error
// Encode renders one part.
Encode(s Sink, p Part) error
// Close writes the dialect's terminator, on success and on failure alike.
Close(s Sink, r *Result) error
}
Encoder renders neutral Parts in a wire dialect.
It is the one seam through which a vendor shape may enter, and it points outward on purpose: this package ships neutral encoders only. The AI SDK UI Message Stream, AG-UI and the OpenAI chat.completion.chunk dialects are vendor response schemas, and a framework that ships one has signed up to track that vendor's changes forever - that format alone changed its framing, its transport, its header name and its semantics inside eighteen months, and its header carries someone else's brand. They belong in iris-contrib, against this interface, pinned to a version.
func NDJSON ¶
func NDJSON() Encoder
NDJSON writes one neutral JSON object per line with no SSE framing.
It is the right choice for a server-to-server consumer: there is no EventSource on the other end, so the framing buys nothing and the parsing is one json.Decoder.
The stream ends with a terminal line, {"type":"done"}, on success and on failure alike, for the same reason the SSE transport has a terminator: its absence is how the consumer detects truncation. A failure after the first byte is reported before it as {"type":"error","error":{...}}, where the object is the rest/errors envelope.
func PlainText ¶
func PlainText() Encoder
PlainText writes the answer's text deltas and nothing else.
It exists for curl and for a client that wants prose rather than a protocol. Reasoning, tool calls and usage are dropped, because there is nowhere in this dialect to put them, and so are errors: there is no terminator and no in-band error frame, because any marker would read as part of the prose. A failure after the first byte still reaches the logger and the collector, but on the wire a truncated answer is indistinguishable from a complete one. Use it where that is acceptable, which is a human reading a terminal.
func SSEJSON ¶
func SSEJSON() Encoder
SSEJSON is the default encoder: one SSE frame per part, named for its kind - text, reasoning, tool, tool-result, usage, error - carrying the neutral JSON of that part. A KindData part keeps the name its producer gave it.
An error part is the exception to "the neutral JSON of that part": its frame carries the rest/errors envelope itself, which is exactly the frame Response.Close writes for a terminal failure, so a client handles one shape for an error whether it was sent mid-stream or ended the stream.
Vendor-neutral and stable, which is the point: a dialect that changes is a dialect that belongs behind this interface rather than inside it.
type Kind ¶
type Kind uint8
Kind identifies what a Part carries.
const ( // KindText is a delta of the answer itself. KindText Kind = iota + 1 // KindReasoning is a delta of a reasoning trace, which many models emit // separately from the answer and most interfaces render differently. KindReasoning // KindToolCall is a request from the model to call a tool. KindToolCall // KindToolResult is what a tool returned. KindToolResult // KindUsage is what the turn cost. KindUsage // KindError is a failure that happened after the response began. KindError // KindData is the open namespace: a producer emitting something this // vocabulary does not model passes it through named rather than losing it. // // It exists because a closed Go union would reject a valid-but-newer // producer, and providers add event types continuously. Strict on emit, // lenient on parse. KindData )
The part vocabulary. Seven kinds, not twenty: a richer grammar of start/delta/end triples only buys compile-time sequence safety to a framework that owns the generation loop, and this one deliberately does not. The id field gives a interface the same correlation for a fraction of the surface.
type Message ¶
type Message struct {
Text string `json:"text,omitempty"`
Reasoning string `json:"reasoning,omitempty"`
// ToolCalls marshal through Part, so the same object shape appears here and
// in a streamed tool frame.
ToolCalls []Part `json:"tool_calls,omitempty"`
Usage *Usage `json:"usage,omitempty"`
}
Message is the non-streaming projection of a turn: what a client that asked for a single document receives.
type Options ¶
type Options struct {
// Stream forces or forbids streaming. Nil negotiates on the request's Accept
// header. Pass &req.Stream to honour an OpenAI-shaped request body.
//
// It is a pointer rather than a bool because the zero value has to be
// distinguishable from "not configured". A plain bool in a struct-valued
// option silently means the opposite of what half of its authors intend.
Stream *bool
// Encoder renders parts on the wire. Nil selects SSEJSON.
Encoder Encoder
// SSE configures the transport when the turn streams over an event stream.
SSE sse.Options
// Coalesce batches consecutive text and reasoning deltas, emitting at most
// one of each per interval. Zero disables it.
//
// It batches parts rather than bytes, because a part carries structure that
// concatenating two encoded frames would destroy.
Coalesce time.Duration
// OnPart runs before each part is encoded. One hook, three outcomes:
//
// (p, nil) forward it, possibly rewritten
// (_, ErrDropPart) drop this part and keep streaming
// (_, any other) fail the turn
//
// This is the whole policy surface, and it is block-only by construction.
// You cannot revise what you have already sent, so the restriction lives in
// the type rather than in a warning: a hook that could rewrite history would
// work in tests and fail on the wire.
//
// It also sees the terminal error that Close writes, as a KindError part, so
// the failure a client is told about goes through the same gate everything
// else does. Returning ErrDropPart there writes no error frame at all, and
// the stream still gets its terminator.
OnPart func(ctx *context.Context, p Part) (Part, error)
// OnDisconnect decides what a vanished client does to the upstream call.
OnDisconnect Disconnect
// OnFinish runs exactly once after the turn ends, under
// context.WithoutCancel, with the assembled answer and the usage. It runs on
// success, on failure and on client disconnect alike, and for a turn the
// handler abandoned without Close (see Result.Abandoned).
//
// Persistence, billing and metrics belong here. The stream is an
// optimisation and never the source of truth: a design in which the only
// copy of the answer is on the wire loses answers.
OnFinish func(ctx stdcontext.Context, r *Result)
// Redact rewrites the client-facing text of an error before it is sent.
//
// It runs last, over text the rest/errors pipeline has already made safe, and
// it exists for the case that pipeline cannot cover on its own: an error the
// application deliberately mapped and described, whose description still
// names a provider, a model or a host that the client has no business
// learning.
//
// A strings.Replacer is the natural implementation and is safe for concurrent
// use. Hold it in an atomic.Pointer if the vocabulary changes at runtime.
//
// It applies to error text only. Rewriting the model's own output is
// OnPart's job, where the decision can be made per part.
Redact func(string) string
// KeepParts retains every part in Result.Parts.
//
// Off by default: a long generation would otherwise be held twice, once as
// assembled text and once as every delta that built it.
KeepParts bool
}
Options configure one turn. Start from DefaultOptions and change what you need: a zero field means "off" rather than "default".
func DefaultOptions ¶
func DefaultOptions() Options
DefaultOptions returns the recommended settings.
type Part ¶
type Part struct {
// contains filtered or unexported fields
}
Part is one increment of a model's answer.
Its fields are unexported and it is built through the constructors below, so an invalid combination - a tool call carrying reasoning text, a usage part with a delta - cannot be constructed, and no encoder has to defend against one. That is as close as Go gets to a sum type, and it turns the grammar into an invariant rather than a doc comment.
func Data ¶
Data returns an event this vocabulary does not model, passed through under its own name.
func Error ¶
Error returns a failure that happened after the response began.
On a streaming turn it is sent through the same policy as the error given to Response.Close: the error is reported through rest/errors, an unmapped or private one is redacted, Options.Redact runs last, and what reaches the wire is the resulting envelope. The text of err itself never does.
func Meter ¶
Meter returns the usage of a turn. It is named Meter rather than Usage so that the constructor and the type do not collide.
func Text ¶
Text returns a delta of the answer. The id correlates deltas that belong to the same message, and may be empty when there is only one.
func ToolCall ¶
ToolCall returns a request to call a tool. args is raw JSON so that a fragment which is not yet valid JSON can still be forwarded.
func ToolResult ¶
ToolResult returns what a tool produced.
func (Part) ArgsAs ¶
ArgsAs decodes the tool arguments of a KindToolCall part into T, under the framework's unmarshal defaults (jsonutil).
func (Part) MarshalJSON ¶
MarshalJSON renders the part in the neutral wire shape. It exists for encoding/json (v1) interop — a downstream logger calling the old json.Marshal on a Part still gets the same bytes.
func (Part) MarshalJSONTo ¶
MarshalJSONTo renders the part in the neutral wire shape, streaming straight into the encoder — the fast path every encoder in this package and ctx.JSON take, with no intermediate []byte.
func (Part) ResultAs ¶
ResultAs decodes the tool result of a KindToolResult part into T, under the framework's unmarshal defaults (jsonutil).
type RelayOptions ¶
type RelayOptions struct {
// Read configures the upstream parser. Set Terminators to []string{"[DONE]"}
// for an OpenAI-dialect upstream.
//
// When Terminators are configured Relay requires one, whatever
// RequireTerminator says: an upstream that closes without its sentinel was
// cut, and the client is told so instead of being handed three deltas and
// a terminator that says the answer is complete. There is no opt-out by
// design; a producer whose stream legitimately ends without the sentinel
// has no sentinel, so leave Terminators empty.
Read sse.ReadOptions
// AllowNonEventStream accepts an upstream that answers 200 with something
// other than text/event-stream, handing the body to the decoder as one
// event instead of failing.
//
// Off by default, because the usual cause is a gateway that returned a JSON
// error with a 200 status, and silently parsing that as an empty stream is
// how a failed generation becomes a blank answer.
AllowNonEventStream bool
}
RelayOptions tune Relay.
type Response ¶
type Response struct {
// contains filtered or unexported fields
}
Response is one turn in progress.
func Open ¶
Open begins a turn.
One handler serves both shapes. When the request asks to stream - through Options.Stream, or through an Accept header naming text/event-stream - parts go out as they are produced. When it does not, they are assembled and written as a single JSON document by Close. Nothing in the handler between those two points changes.
Like sse.Open it writes nothing: an authorization failure or a policy veto after Open still produces an ordinary status code with an ordinary body.
func (*Response) Close ¶
Close ends the turn: it flushes what is pending, writes the terminator or the assembled document, and runs OnFinish exactly once.
When err is non-nil and nothing has reached the client, Close writes nothing and returns err, so the caller renders an ordinary HTTP error. After the first byte the status code is spent, so the failure is reported in-band in whatever form the encoder has: an error frame on SSE, an error line on NDJSON, and nothing on PlainText, which has no room for one. In every case it goes through rest/errors first, so the logger and the collector hear of it.
A streaming turn with no parts at all still commits the response and writes the terminator, as the non-streaming shape writes an empty document: an empty answer is an answer, and a client must be able to tell it from a connection that dropped before the first byte.
func (*Response) Committed ¶
Committed reports whether any byte has reached the client. A failover ladder must consult it before retrying an upstream: after the first delta, a second attempt splices two answers into one response.
func (*Response) Context ¶
func (r *Response) Context() stdcontext.Context
Context is done when the client disconnects or the turn ends. It is nil-safe to select on for a non-streaming turn, where it follows the request.
func (*Response) Result ¶
Result returns a snapshot of what the turn produced so far. It is complete after Close.
It is a copy, taken under the lock: Close keeps writing the turn's own record until OnFinish has run, and a pointer into it would race those writes. The ToolCalls and Parts slices are shared with the turn and are read-only.
func (*Response) Send ¶
Send renders one part.
A KindError part on a streaming turn goes through the same policy as the error given to Close - reported through rest/errors, redacted unless deliberately mapped, rewritten by Options.Redact - and reaches the wire as the resulting envelope. Result.Err keeps the original. When the client has already gone there is nothing to report, and the part is dropped.
type Result ¶
type Result struct {
// Streaming reports which shape the turn took.
Streaming bool
// Committed reports whether any byte reached the client. It is the line
// between an error response and a truncated answer.
Committed bool
// Disconnected reports that the client went away before the turn ended.
Disconnected bool
// Abandoned reports that the handler returned without calling Close. The
// framework ended the turn before the response was recycled: nothing was
// written after the handler returned, a part still pending was dropped,
// and the client got no terminator. Committed says whether anything had
// reached the client before that.
Abandoned bool
// Text is the assembled answer.
Text string
// Reasoning is the assembled reasoning trace.
Reasoning string
// ToolCalls are the tool calls the model asked for, as the parts that
// carried them: a tool-call part already holds the id, the name and the raw
// arguments, so a second type for the same data would only be one more thing
// to keep in step.
ToolCalls []Part
// Usage is what the turn cost, accumulated across every usage part.
Usage Usage
// Parts is every part, when Options.KeepParts is set.
Parts []Part
// Err is why the turn ended badly, or nil.
Err error
// TTFP is the time to the first part, which is the latency a reader
// actually perceives. Total duration is dominated by how long the answer is.
TTFP time.Duration
// Duration is the whole turn.
Duration time.Duration
}
Result is what a turn produced. It is handed to OnFinish and available from Response.Result.
type Sink ¶
type Sink interface {
// Frame writes one SSE frame, or returns ErrNotFramed.
Frame(ev sse.Event) error
// Write writes raw bytes, or returns ErrNotRaw.
io.Writer
// Flush pushes whatever has been written to the client.
Flush() error
}
Sink is what an Encoder writes into.
An encoder uses whichever half matches the transport it asked for through ContentType: an empty content type gets an event stream and uses Frame, and anything else gets a plain streaming response and uses Write.
type TagSplitter ¶
type TagSplitter struct {
// contains filtered or unexported fields
}
TagSplitter separates text wrapped in an inline tag from the rest, across chunk boundaries.
Several models emit their reasoning inline, fenced by a tag such as <think>, rather than in a separate field. The tag arrives split across deltas as often as not - "<th" in one chunk and "ink>" in the next - so a naive strings.Contains on each chunk both misses the tag and leaks its characters into the answer.
It holds a small buffer, not a stream: feed it deltas, take the parts it returns. Use it through the pointer TagSplit returns; it is not safe for concurrent use.
func TagSplit ¶
func TagSplit(id, open, close string) *TagSplitter
TagSplit returns a splitter for text fenced by open and close, emitting the fenced text as reasoning and everything else as answer text.
splitter := ai.TagSplit(msgID, "<think>", "</think>")
for _, p := range splitter.Write(delta) {
res.Send(p)
}
// ... at the end of the stream:
for _, p := range splitter.Flush() {
res.Send(p)
}
It panics on an empty marker: an empty string is found at every position, so the splitter would loop forever or call the whole answer reasoning, and no model emits an invisible tag.
func (*TagSplitter) Flush ¶
func (s *TagSplitter) Flush() []Part
Flush releases whatever is still held back, at the end of a stream.
A tag that never completed is not a tag: its characters are part of the answer, and dropping them would silently truncate it.
func (*TagSplitter) Inside ¶
func (s *TagSplitter) Inside() bool
Inside reports whether the splitter is currently inside the tag.
func (*TagSplitter) StartInside ¶
func (s *TagSplitter) StartInside() *TagSplitter
StartInside tells the splitter that the stream begins inside the tag.
It exists for models that emit a closing tag without ever having emitted an opening one, which is a real and otherwise baffling dialect: everything before the first </think> is reasoning.
func (*TagSplitter) Write ¶
func (s *TagSplitter) Write(delta string) []Part
Write feeds one delta and returns the parts it completed.
It holds back only what could still turn out to be the start of a tag, so a tag split across any number of chunks is recognised, and at most len(tag)-1 characters are ever delayed.
type UpstreamError ¶
type UpstreamError struct {
// Status is the HTTP status code, or 0 when the request never got one.
Status int
// Class is what to do about it.
Class Class
// Message is the provider's own message, when the body carried one in any of
// the shapes providers use.
Message string
// Code is the provider's machine-readable code, when there was one.
Code string
// RequestID is the provider's request identifier, from the body or from one
// of the usual headers. It is the only thing that makes a support
// conversation about a single failed generation possible.
RequestID string
// RetryAfter is the delay the provider asked for, from Retry-After.
RetryAfter time.Duration
// Body is the response body, truncated. Nil when there was no response.
Body []byte
// contains filtered or unexported fields
}
UpstreamError is a typed failure from a provider.
func AsUpstream ¶
func AsUpstream(err error) (*UpstreamError, bool)
AsUpstream extracts an *UpstreamError from an error chain.
func Classify ¶
func Classify(resp *http.Response, err error, opts ...ClassifyOptions) *UpstreamError
Classify turns an upstream exchange into a typed failure, or nil when nothing went wrong.
It takes the response and not only the error, and that is the whole point. A well-behaved Go HTTP client returns a non-2xx response with a nil error, so a Classify(err) signature would have to recover the status code from a string and would recreate exactly the type erasure it exists to delete.
It reads and closes the response body on failure, keeping up to MaxBody of it, because the body is where the provider put the reason. On success it leaves the body untouched for the caller to stream.
func (*UpstreamError) ErrorCode ¶
func (e *UpstreamError) ErrorCode() errors.ErrorCodeName
ErrorCode returns the canonical rest/errors code for this failure.
The mapping is deliberate, and it is not a passthrough of the upstream's own status. A provider 401 means this gateway's credential is finished, which is not the caller's problem to fix; telling them they are unauthorised sends them to check a key they do not own.
It returns an empty code for a cancelled request, which means there is nothing to report: the caller has gone, so nothing should be written and nobody should be paged. Test it with ErrorCodeName.String() == "", the same sentinel the errors package uses elsewhere.
func (*UpstreamError) HTTPStatus ¶
func (e *UpstreamError) HTTPStatus() int
HTTPStatus is the status code to give the client for this failure.
It is not the upstream's status code, and the difference matters. A 401 from a provider means the gateway's credential is finished, which is a 502 to the caller: telling them they are unauthorised would send them off to check their own key. The one code that does pass through is 429, because rate limiting is the caller's problem too and the delay is worth forwarding.
func (*UpstreamError) Report ¶
func (e *UpstreamError) Report(ctx *context.Context) *errors.Error
Report runs this failure through the rest/errors pipeline under its canonical code, returning the client-safe Error, or nil when there is nothing to report.
The client is told what the failure means, never what the provider said. A provider's message is text this service did not write, about a system the caller cannot see, and it routinely names the vendor, the model or the host; forwarding it is a leak whatever the status code was. It goes to the logger and the collector, where it is useful, and the caller gets a sentence derived from the class.
The error is wrapped as private on the way in, so even if this function's own text were removed the pipeline would still redact rather than forward.
func (*UpstreamError) Retryable ¶
func (e *UpstreamError) Retryable() bool
Retryable reports whether trying again could plausibly help.
func (*UpstreamError) Timeout ¶
func (e *UpstreamError) Timeout() bool
Timeout reports whether the request failed because the upstream did not answer in time: a context deadline, or a transport error whose Timeout() is true. net/http's own timeout errors - http.Client.Timeout, Transport.ResponseHeaderTimeout - are both at once; the net.Error check is for timeouts from elsewhere, such as a net.Conn read deadline, which matches os.ErrDeadlineExceeded and not context.DeadlineExceeded. It is transient - a stalled provider is the failure a gateway sees most - and it is reported to the client as DEADLINE_EXCEEDED.
func (*UpstreamError) Unwrap ¶
func (e *UpstreamError) Unwrap() error
Unwrap exposes the transport error, so errors.Is against context.Canceled and the net package's own errors keeps working.
type Usage ¶
type Usage struct {
// omitzero, not omitempty: under encoding/json/v2 omitempty no longer
// omits a zero number.
InputTokens int64 `json:"input_tokens,omitzero"`
OutputTokens int64 `json:"output_tokens,omitzero"`
CachedInputTokens int64 `json:"cached_input_tokens,omitzero"`
ReasoningTokens int64 `json:"reasoning_tokens,omitzero"`
// Extra carries counters a provider reports that this struct does not name,
// so nothing is lost in translation.
Extra map[string]int64 `json:"extra,omitempty"`
}
Usage is what one turn cost.
Cost in money is deliberately absent: pricing a provider means tracking every provider's price list forever, and it changes more often than this package would ship. Compute it in OnFinish, where you already know which model ran.