ai

package
v14.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 12, 2026 License: BSD-3-Clause, BSD-3-Clause Imports: 17 Imported by: 0

README

AI backends

Renders and relays AI model streams over HTTP.

The package is transport-aware and provider-agnostic. It understands the shapes that streamed generation takes, including text deltas, reasoning traces, tool-call fragments, usage trailers and an error that arrives after the response has already begun. It ships no vendor SDK, no HTTP client for any AI service, and no vendor request or response schema. What travels over the wire is your business; the shape of the wire is this package's.

go get github.com/kataras/iris/v14@latest
import (
	"github.com/kataras/iris/v14/ai"
	"github.com/kataras/iris/v14/rest/errors"
)

One handler, both shapes

func chat(ctx iris.Context) {
	req, ok := ctx.BindJSON[ChatRequest]()
	if !ok {
		return
	}

	o := ai.DefaultOptions()
	o.Stream = &req.Stream // honour the request body, as an OpenAI-shaped API must.

	res, err := ai.Open(ctx, o)
	if err != nil {
		errors.HandleError(ctx, err) // renders the 500 envelope, logs the cause.
		return
	}

	for delta := range generate(req) {
		if err := res.Send(ai.Text("m1", delta)); err != nil {
			res.Close(err)
			return
		}
	}

	res.Close(nil)
}

When the request asks to stream, the parts go out as they are produced. When it does not, they are assembled and written as one JSON document. Nothing between Open and Close changes. Options.Stream is a *bool rather than a bool because the zero value has to stay distinguishable from "not configured".

Like sse.Open, ai.Open writes nothing, so an authorization failure or a policy veto after it still produces an ordinary status code with an ordinary body.

The part vocabulary

Seven kinds, and constructors instead of a struct literal:

ai.Text(id, delta)
ai.Reasoning(id, delta)
ai.ToolCall(id, name, args)
ai.ToolResult(id, result)
ai.Meter(usage)
ai.Error(err)
ai.Data(name, raw) // the open namespace

Part has unexported fields, so an invalid combination such as a tool call carrying reasoning text cannot be constructed and no encoder has to defend against one. That turns the grammar into an invariant rather than a doc comment.

KindData exists because a closed union would reject a valid-but-newer producer, and providers add event types continuously. Strict on emit, lenient on parse.

Encoders

SSEJSON is the default: one event per part, named for its kind, carrying neutral JSON. NDJSON writes one object per line for a server-to-server consumer, and ends with a terminal line, {"type":"done"}, so the consumer can detect truncation the way an SSE client does. PlainText writes the answer and nothing else, for curl: no terminator, no in-band error, because any marker would read as prose.

A streaming turn with no parts at all still commits and writes its terminator, as the non-streaming shape writes {}: an empty answer must be distinguishable from a connection that dropped before the first byte.

The Encoder interface is the single seam through which a vendor dialect may enter, and it points outward. The AI SDK UI Message Stream, AG-UI and the OpenAI chat.completion.chunk shapes are vendor response schemas; a framework that ships one has signed up to track that vendor forever. That format alone changed its framing, its transport, its header name and its semantics inside eighteen months, and its header carries someone else's brand. Those belong in iris-contrib, against this interface, pinned to a version.

Relaying an upstream

resp, err := httpClient.Do(req)
if ue := ai.Classify(resp, err); ue != nil {
	res.Close(ue)
	return
}

res.Close(ai.Relay(res, resp, decoder, ai.RelayOptions{
	Read: sse.ReadOptions{Terminators: []string{"[DONE]"}},
}))

Relay classifies the response first, so a non-2xx upstream comes back as a typed error rather than as an empty stream, and it checks the content type, so a gateway that answers 200 with a JSON error does not silently become a blank answer. An upstream cut mid-event, or closed without the [DONE] it was configured to send, is an error too (matching io.ErrUnexpectedEOF), so the client gets an error frame rather than a terminator that says the answer was complete.

The Decoder is your code. The framework has no business knowing what a particular vendor calls a delta this month.

Typed upstream failures

Classify takes the response and not only the error, and that is the whole point. A well-behaved Go HTTP client returns a non-2xx response with a nil error, so a Classify(err) form would have to recover the status code from a string and would recreate exactly the type erasure it exists to delete. The alternative is what AI gateways actually end up with: a regular expression over err.Error().

Class Meaning
ClassFatal The request is wrong and would be just as wrong on another credential.
ClassTransient A 5xx, a reset, a dial error, a timeout. Retrying is reasonable. Timeout() tells a stalled provider apart from the rest; it is reported as DEADLINE_EXCEEDED.
ClassRateLimited A 429. Its own class because it is a routing signal, not a health signal: a credential pool that counts it as a fault quarantines a healthy key for being popular.
ClassExhausted This credential or account cannot serve the request at all. A different one might.
ClassCanceled The caller went away (context.Canceled, and only that). A retry spends tokens on an answer nobody is waiting for.

401 and 403 map to exhaustion rather than fatal, because in a gateway they almost always mean this credential is finished rather than this request is malformed, and treating them as fatal is how a rotated key takes the whole service down instead of failing over.

What the client is told

Every failure, whether it arrives before the first byte or after it, and whether it ends the turn through Close(err) or is sent as a part with ai.Error(err), goes through rest/errors and reaches the client as the same envelope:

{"id":"5d09cc97-...","http_error_code":{"canonical_name":"RESOURCE_EXHAUSTED","status":429},
 "message":"too many requests","detail":"the service is rate limited, try again shortly"}

Two things happen there that you do not have to write.

The failure is reported. The logger and the collector fire with the real error, which is what carries it to a cloud sink. errors.LogError is the hook, so an existing setup keeps working unchanged. Without this a mid-stream failure is invisible: the render path cannot run once a status code is committed, and skipping the pipeline skips the reporting with it.

The provider's words never reach the client. A provider message is text this service did not write, about a system the caller cannot see, and it routinely names the vendor, the model or the host. It goes to the logger, where it is useful. The caller gets a sentence derived from the class, and a classified failure keeps its canonical code, so a rate limit is RESOURCE_EXHAUSTED rather than a generic internal error.

On SSE the envelope is the data of the error event. On NDJSON it is nested, {"type":"error","error":{...}}, because every line carries a type. On PlainText nothing is written: the logger and the collector still hear of the failure, the reader does not.

Only the client's own disconnect silences the report. An upstream timeout looks like context.DeadlineExceeded to errors.Is, and it is still classified, reported and written, because the client is waiting to hear about it.

For the remaining case, an error your application mapped and described on purpose whose description still names a vendor, Options.Redact rewrites the client-facing text:

o.Redact = strings.NewReplacer("anthropic", "the model provider").Replace

A strings.Replacer is safe for concurrent use; hold it in an atomic.Pointer if the vocabulary changes at runtime. Redact applies to error text only. Rewriting the model's own output is OnPart's job, where the decision can be made per part.

OnPart also sees the terminal error, as a KindError part, so one hook governs everything the client is told. Returning ErrDropPart there writes no error frame at all, and the stream still gets its terminator.

Reasoning fenced in a tag

Several models emit their reasoning inline, fenced by a tag such as <think>. The tag arrives split across deltas as often as not, so a strings.Contains on each chunk both misses it and leaks its characters into the answer.

splitter := ai.TagSplit("m1", "<think>", "</think>")

for _, p := range splitter.Write(delta) {
	res.Send(p)
}
// at the end of the stream:
for _, p := range splitter.Flush() {
	res.Send(p)
}

StartInside() covers models that emit a closing tag having never emitted an opening one.

Policy and accounting

Options.OnPart runs before each part is encoded, with three outcomes: forward it, possibly rewritten; return ErrDropPart to drop it and keep going; return anything else to fail the turn. It is block-only by construction, because you cannot revise what you have already sent, and that restriction lives in the type rather than in a warning.

Options.OnFinish runs exactly once after the turn ends, under context.WithoutCancel, with the assembled answer and the usage. It runs on success, on failure and on client disconnect alike, and for a turn the handler abandoned without Close, which the framework ends before the response is recycled and marks with Result.Abandoned. Persistence, billing and metrics belong there. The stream is an optimisation and never the source of truth: a design in which the only copy of the answer is on the wire loses answers.

What this package deliberately does not do

  • An agent loop. Turns, stopping conditions and memory vary too much per application. Every framework that shipped one has spent the years since adding kill switches to it. The primitives here make a manual loop a few lines.
  • Any LLM client, model catalog, token counter or cost table. client.Client is all the HTTP a relay needs, and pricing a provider means tracking every provider's price list forever.
  • A conversation, memory or vector store. OnFinish hands you the assembled turn and the usage; the schema is yours.
  • Vendor wire encoders. See above.
  • OpenTelemetry attributes. The GenAI conventions left the main semantic-conventions repository and everything in the new home is still marked development, having already renamed its central attribute once. Result carries TTFP, Duration and Usage; emit the spans you want, pinned to the version you chose.

Examples

_examples/05-response/streaming-ai and _examples/16-advanced/ai-gateway

Documentation

Overview

Package ai renders and relays AI model streams over HTTP.

It is transport-aware and provider-agnostic. It understands the shapes that streamed generation takes - text deltas, reasoning traces, tool-call fragments, usage trailers, an error that arrives after the response has already begun - and it ships no vendor SDK, no HTTP client for any AI service, and no vendor request or response schema. What travels over the wire is your business; the shape of the wire is this package's.

The core is a small neutral vocabulary, Part, and a pluggable Encoder that renders it in a wire dialect. One handler serves both a streaming and a non-streaming request, because that is what every OpenAI-shaped API has to do and nothing in Go offers it.

It deliberately does not include an agent loop, a conversation store, a prompt template format, a token counter or a cost table. Those vary too much between applications to belong in a framework, and every framework that shipped one has spent the years since adding escape hatches to it.

Index

Constants

View Source
const DefaultMaxErrorBody = 64 << 10

DefaultMaxErrorBody caps how much of a failed response Classify keeps.

Variables

View Source
var ErrDropPart = errors.New("iris: ai: drop part")

ErrDropPart tells Open to discard the current part and keep going. Return it from Options.OnPart.

View Source
var ErrNonEventStream = errors.New("iris: ai: upstream response is not an event stream")

ErrNonEventStream reports that an upstream answered 200 with a body that is not an event stream.

View Source
var ErrNotFramed = errors.New("iris: ai: sink is not an event stream")

ErrNotFramed reports that a Sink does not carry SSE frames, which is the case for a line-oriented dialect such as NDJSON.

View Source
var ErrNotRaw = errors.New("iris: ai: sink takes frames, not raw bytes")

ErrNotRaw reports that a Sink does not take raw bytes, which is the case for an SSE dialect, where everything must be a frame.

Functions

func Relay

func Relay(r *Response, resp *http.Response, dec Decoder, opts ...RelayOptions) error

Relay reads an upstream event stream, decodes it and forwards the parts to r.

It closes resp.Body. It classifies the response first, so a non-2xx upstream comes back as a typed *UpstreamError rather than as an empty stream, and it checks the content type, so a gateway that answers 200 with a JSON error does not silently become a blank answer. A failed read of the upstream body - a reset, a timeout, a stream cut mid-event or closed without the sentinel RelayOptions.Read.Terminators promised - comes back through Classify as a transient *UpstreamError wrapping the cause (io.ErrUnexpectedEOF for the last two), so the client gets a BAD_GATEWAY error frame rather than a terminator claiming the answer was complete; a client that hangs up mid-read comes back as ClassCanceled. Decoder errors are returned as they are: they are the application's own.

resp, err := httpClient.Do(req)
if ue := ai.Classify(resp, err); ue != nil {
	return res.Close(ue)
}

return res.Close(ai.Relay(res, resp, decoder, ai.RelayOptions{
	Read: sse.ReadOptions{Terminators: []string{"[DONE]"}},
}))

Types

type Class

type Class uint8

Class is what a caller actually needs to decide from an upstream failure: whether to retry, whether to try a different credential, and whether to give up.

It exists because the alternative is what every AI gateway ends up with - a regular expression over err.Error(), because the status code was flattened into a string three packages ago and there is nowhere else left to read it.

const (
	// ClassNone means the response was fine.
	ClassNone Class = iota

	// ClassFatal is a request-shaped failure: the request is wrong and would be
	// just as wrong on another credential or another model. Retrying spends
	// money to get the same answer.
	ClassFatal

	// ClassTransient is a failure that may not recur: a 5xx, a reset, a dial
	// error. Retrying is reasonable.
	ClassTransient

	// ClassRateLimited is a 429. It is deliberately its own class rather than a
	// kind of exhaustion: it is a routing signal, not a health signal, and a
	// credential pool that counts it as a fault will quarantine a perfectly
	// healthy key for being popular.
	ClassRateLimited

	// ClassExhausted means this credential or account cannot serve the request
	// at all: it is unauthorised, out of quota, or suspended. A different
	// credential might work; this one will not, for a while.
	ClassExhausted

	// ClassCanceled means the caller went away. It is not a failure of the
	// upstream, and a second attempt would spend tokens on a response nobody is
	// waiting for.
	ClassCanceled
)

func (Class) Retryable

func (c Class) Retryable() bool

Retryable reports whether trying again could plausibly help.

func (Class) String

func (c Class) String() string

String returns a stable, lowercase name, suitable for a metric label.

type ClassifyOptions

type ClassifyOptions struct {
	// MaxBody caps how much of the response body is read and kept. Default
	// DefaultMaxErrorBody.
	MaxBody int64
}

ClassifyOptions tune Classify.

type Decoder

type Decoder interface {
	Decode(ev sse.Event) ([]Part, error)
}

Decoder turns one upstream event into zero or more neutral parts.

This is where a provider's dialect is understood, and it is the caller's code rather than the framework's: the framework has no business knowing what a particular vendor calls a delta this month.

type DecoderFunc

type DecoderFunc func(ev sse.Event) ([]Part, error)

DecoderFunc adapts a function to Decoder.

func (DecoderFunc) Decode

func (f DecoderFunc) Decode(ev sse.Event) ([]Part, error)

Decode implements Decoder.

type Disconnect

type Disconnect uint8

Disconnect decides what a vanished client does to the work in progress.

const (
	// DisconnectCancel ends the turn as soon as the client goes away. It is the
	// default because an abandoned generation burns tokens nobody will read.
	DisconnectCancel Disconnect = iota

	// DisconnectDetach keeps consuming the upstream after the client has gone,
	// so OnFinish still records what was actually spent.
	//
	// It does not mean the handler returns and work continues in the background.
	// The handler still runs to completion; only the client is gone.
	DisconnectDetach
)

type Encoder

type Encoder interface {
	// ContentType is the response media type. Empty means text/event-stream and
	// selects the SSE transport.
	ContentType() string
	// Header applies any dialect headers before anything is written.
	Header(h http.Header)
	// Open is called once, before the first part.
	Open(s Sink) error
	// Encode renders one part.
	Encode(s Sink, p Part) error
	// Close writes the dialect's terminator, on success and on failure alike.
	Close(s Sink, r *Result) error
}

Encoder renders neutral Parts in a wire dialect.

It is the one seam through which a vendor shape may enter, and it points outward on purpose: this package ships neutral encoders only. The AI SDK UI Message Stream, AG-UI and the OpenAI chat.completion.chunk dialects are vendor response schemas, and a framework that ships one has signed up to track that vendor's changes forever - that format alone changed its framing, its transport, its header name and its semantics inside eighteen months, and its header carries someone else's brand. They belong in iris-contrib, against this interface, pinned to a version.

func NDJSON

func NDJSON() Encoder

NDJSON writes one neutral JSON object per line with no SSE framing.

It is the right choice for a server-to-server consumer: there is no EventSource on the other end, so the framing buys nothing and the parsing is one json.Decoder.

The stream ends with a terminal line, {"type":"done"}, on success and on failure alike, for the same reason the SSE transport has a terminator: its absence is how the consumer detects truncation. A failure after the first byte is reported before it as {"type":"error","error":{...}}, where the object is the rest/errors envelope.

func PlainText

func PlainText() Encoder

PlainText writes the answer's text deltas and nothing else.

It exists for curl and for a client that wants prose rather than a protocol. Reasoning, tool calls and usage are dropped, because there is nowhere in this dialect to put them, and so are errors: there is no terminator and no in-band error frame, because any marker would read as part of the prose. A failure after the first byte still reaches the logger and the collector, but on the wire a truncated answer is indistinguishable from a complete one. Use it where that is acceptable, which is a human reading a terminal.

func SSEJSON

func SSEJSON() Encoder

SSEJSON is the default encoder: one SSE frame per part, named for its kind - text, reasoning, tool, tool-result, usage, error - carrying the neutral JSON of that part. A KindData part keeps the name its producer gave it.

An error part is the exception to "the neutral JSON of that part": its frame carries the rest/errors envelope itself, which is exactly the frame Response.Close writes for a terminal failure, so a client handles one shape for an error whether it was sent mid-stream or ended the stream.

Vendor-neutral and stable, which is the point: a dialect that changes is a dialect that belongs behind this interface rather than inside it.

type Kind

type Kind uint8

Kind identifies what a Part carries.

const (
	// KindText is a delta of the answer itself.
	KindText Kind = iota + 1
	// KindReasoning is a delta of a reasoning trace, which many models emit
	// separately from the answer and most interfaces render differently.
	KindReasoning
	// KindToolCall is a request from the model to call a tool.
	KindToolCall
	// KindToolResult is what a tool returned.
	KindToolResult
	// KindUsage is what the turn cost.
	KindUsage
	// KindError is a failure that happened after the response began.
	KindError
	// KindData is the open namespace: a producer emitting something this
	// vocabulary does not model passes it through named rather than losing it.
	//
	// It exists because a closed Go union would reject a valid-but-newer
	// producer, and providers add event types continuously. Strict on emit,
	// lenient on parse.
	KindData
)

The part vocabulary. Seven kinds, not twenty: a richer grammar of start/delta/end triples only buys compile-time sequence safety to a framework that owns the generation loop, and this one deliberately does not. The id field gives a interface the same correlation for a fraction of the surface.

func (Kind) String

func (k Kind) String() string

String returns the wire name of the kind, which is also the SSE event name the default encoder uses.

type Message

type Message struct {
	Text      string `json:"text,omitempty"`
	Reasoning string `json:"reasoning,omitempty"`
	// ToolCalls marshal through Part, so the same object shape appears here and
	// in a streamed tool frame.
	ToolCalls []Part `json:"tool_calls,omitempty"`
	Usage     *Usage `json:"usage,omitempty"`
}

Message is the non-streaming projection of a turn: what a client that asked for a single document receives.

type Options

type Options struct {
	// Stream forces or forbids streaming. Nil negotiates on the request's Accept
	// header. Pass &req.Stream to honour an OpenAI-shaped request body.
	//
	// It is a pointer rather than a bool because the zero value has to be
	// distinguishable from "not configured". A plain bool in a struct-valued
	// option silently means the opposite of what half of its authors intend.
	Stream *bool

	// Encoder renders parts on the wire. Nil selects SSEJSON.
	Encoder Encoder

	// SSE configures the transport when the turn streams over an event stream.
	SSE sse.Options

	// Coalesce batches consecutive text and reasoning deltas, emitting at most
	// one of each per interval. Zero disables it.
	//
	// It batches parts rather than bytes, because a part carries structure that
	// concatenating two encoded frames would destroy.
	Coalesce time.Duration

	// OnPart runs before each part is encoded. One hook, three outcomes:
	//
	//	(p, nil)          forward it, possibly rewritten
	//	(_, ErrDropPart)  drop this part and keep streaming
	//	(_, any other)    fail the turn
	//
	// This is the whole policy surface, and it is block-only by construction.
	// You cannot revise what you have already sent, so the restriction lives in
	// the type rather than in a warning: a hook that could rewrite history would
	// work in tests and fail on the wire.
	//
	// It also sees the terminal error that Close writes, as a KindError part, so
	// the failure a client is told about goes through the same gate everything
	// else does. Returning ErrDropPart there writes no error frame at all, and
	// the stream still gets its terminator.
	OnPart func(ctx *context.Context, p Part) (Part, error)

	// OnDisconnect decides what a vanished client does to the upstream call.
	OnDisconnect Disconnect

	// OnFinish runs exactly once after the turn ends, under
	// context.WithoutCancel, with the assembled answer and the usage. It runs on
	// success, on failure and on client disconnect alike, and for a turn the
	// handler abandoned without Close (see Result.Abandoned).
	//
	// Persistence, billing and metrics belong here. The stream is an
	// optimisation and never the source of truth: a design in which the only
	// copy of the answer is on the wire loses answers.
	OnFinish func(ctx stdcontext.Context, r *Result)

	// Redact rewrites the client-facing text of an error before it is sent.
	//
	// It runs last, over text the rest/errors pipeline has already made safe, and
	// it exists for the case that pipeline cannot cover on its own: an error the
	// application deliberately mapped and described, whose description still
	// names a provider, a model or a host that the client has no business
	// learning.
	//
	// A strings.Replacer is the natural implementation and is safe for concurrent
	// use. Hold it in an atomic.Pointer if the vocabulary changes at runtime.
	//
	// It applies to error text only. Rewriting the model's own output is
	// OnPart's job, where the decision can be made per part.
	Redact func(string) string

	// KeepParts retains every part in Result.Parts.
	//
	// Off by default: a long generation would otherwise be held twice, once as
	// assembled text and once as every delta that built it.
	KeepParts bool
}

Options configure one turn. Start from DefaultOptions and change what you need: a zero field means "off" rather than "default".

func DefaultOptions

func DefaultOptions() Options

DefaultOptions returns the recommended settings.

type Part

type Part struct {
	// contains filtered or unexported fields
}

Part is one increment of a model's answer.

Its fields are unexported and it is built through the constructors below, so an invalid combination - a tool call carrying reasoning text, a usage part with a delta - cannot be constructed, and no encoder has to defend against one. That is as close as Go gets to a sum type, and it turns the grammar into an invariant rather than a doc comment.

func Data

func Data(name string, raw jsontext.Value) Part

Data returns an event this vocabulary does not model, passed through under its own name.

func Error

func Error(err error) Part

Error returns a failure that happened after the response began.

On a streaming turn it is sent through the same policy as the error given to Response.Close: the error is reported through rest/errors, an unmapped or private one is redacted, Options.Redact runs last, and what reaches the wire is the resulting envelope. The text of err itself never does.

func Meter

func Meter(u Usage) Part

Meter returns the usage of a turn. It is named Meter rather than Usage so that the constructor and the type do not collide.

func Reasoning

func Reasoning(id, delta string) Part

Reasoning returns a delta of a reasoning trace.

func Text

func Text(id, delta string) Part

Text returns a delta of the answer. The id correlates deltas that belong to the same message, and may be empty when there is only one.

func ToolCall

func ToolCall(id, name string, args jsontext.Value) Part

ToolCall returns a request to call a tool. args is raw JSON so that a fragment which is not yet valid JSON can still be forwarded.

func ToolResult

func ToolResult(id string, result jsontext.Value) Part

ToolResult returns what a tool produced.

func (Part) Args

func (p Part) Args() jsontext.Value

Args is the raw tool arguments for KindToolCall.

func (Part) ArgsAs

func (p Part) ArgsAs[T any]() (T, error)

ArgsAs decodes the tool arguments of a KindToolCall part into T, under the framework's unmarshal defaults (jsonutil).

func (Part) Delta

func (p Part) Delta() string

Delta is the text increment for KindText and KindReasoning.

func (Part) Err

func (p Part) Err() error

Err is the failure, for KindError.

func (Part) ID

func (p Part) ID() string

ID is the correlation id, for the kinds that have one.

func (Part) IsZero

func (p Part) IsZero() bool

IsZero reports whether this part was never constructed.

func (Part) Kind

func (p Part) Kind() Kind

Kind reports what this part carries.

func (Part) MarshalJSON

func (p Part) MarshalJSON() ([]byte, error)

MarshalJSON renders the part in the neutral wire shape. It exists for encoding/json (v1) interop — a downstream logger calling the old json.Marshal on a Part still gets the same bytes.

func (Part) MarshalJSONTo

func (p Part) MarshalJSONTo(enc *jsontext.Encoder) error

MarshalJSONTo renders the part in the neutral wire shape, streaming straight into the encoder — the fast path every encoder in this package and ctx.JSON take, with no intermediate []byte.

func (Part) Name

func (p Part) Name() string

Name is the tool name for KindToolCall and the event name for KindData.

func (Part) Raw

func (p Part) Raw() jsontext.Value

Raw is the passthrough payload, for KindData.

func (Part) Result

func (p Part) Result() jsontext.Value

Result is the raw tool result for KindToolResult.

func (Part) ResultAs

func (p Part) ResultAs[T any]() (T, error)

ResultAs decodes the tool result of a KindToolResult part into T, under the framework's unmarshal defaults (jsonutil).

func (Part) String

func (p Part) String() string

String is for logs and test failures, not for the wire.

func (Part) Usage

func (p Part) Usage() Usage

Usage is what the turn cost, for KindUsage.

type RelayOptions

type RelayOptions struct {
	// Read configures the upstream parser. Set Terminators to []string{"[DONE]"}
	// for an OpenAI-dialect upstream.
	//
	// When Terminators are configured Relay requires one, whatever
	// RequireTerminator says: an upstream that closes without its sentinel was
	// cut, and the client is told so instead of being handed three deltas and
	// a terminator that says the answer is complete. There is no opt-out by
	// design; a producer whose stream legitimately ends without the sentinel
	// has no sentinel, so leave Terminators empty.
	Read sse.ReadOptions

	// AllowNonEventStream accepts an upstream that answers 200 with something
	// other than text/event-stream, handing the body to the decoder as one
	// event instead of failing.
	//
	// Off by default, because the usual cause is a gateway that returned a JSON
	// error with a 200 status, and silently parsing that as an empty stream is
	// how a failed generation becomes a blank answer.
	AllowNonEventStream bool
}

RelayOptions tune Relay.

type Response

type Response struct {
	// contains filtered or unexported fields
}

Response is one turn in progress.

func Open

func Open(ctx *context.Context, opts ...Options) (*Response, error)

Open begins a turn.

One handler serves both shapes. When the request asks to stream - through Options.Stream, or through an Accept header naming text/event-stream - parts go out as they are produced. When it does not, they are assembled and written as a single JSON document by Close. Nothing in the handler between those two points changes.

Like sse.Open it writes nothing: an authorization failure or a policy veto after Open still produces an ordinary status code with an ordinary body.

func (*Response) Close

func (r *Response) Close(err error) error

Close ends the turn: it flushes what is pending, writes the terminator or the assembled document, and runs OnFinish exactly once.

When err is non-nil and nothing has reached the client, Close writes nothing and returns err, so the caller renders an ordinary HTTP error. After the first byte the status code is spent, so the failure is reported in-band in whatever form the encoder has: an error frame on SSE, an error line on NDJSON, and nothing on PlainText, which has no room for one. In every case it goes through rest/errors first, so the logger and the collector hear of it.

A streaming turn with no parts at all still commits the response and writes the terminator, as the non-streaming shape writes an empty document: an empty answer is an answer, and a client must be able to tell it from a connection that dropped before the first byte.

func (*Response) Committed

func (r *Response) Committed() bool

Committed reports whether any byte has reached the client. A failover ladder must consult it before retrying an upstream: after the first delta, a second attempt splices two answers into one response.

func (*Response) Context

func (r *Response) Context() stdcontext.Context

Context is done when the client disconnects or the turn ends. It is nil-safe to select on for a non-streaming turn, where it follows the request.

func (*Response) Result

func (r *Response) Result() Result

Result returns a snapshot of what the turn produced so far. It is complete after Close.

It is a copy, taken under the lock: Close keeps writing the turn's own record until OnFinish has run, and a pointer into it would race those writes. The ToolCalls and Parts slices are shared with the turn and are read-only.

func (*Response) Send

func (r *Response) Send(p Part) error

Send renders one part.

A KindError part on a streaming turn goes through the same policy as the error given to Close - reported through rest/errors, redacted unless deliberately mapped, rewritten by Options.Redact - and reaches the wire as the resulting envelope. Result.Err keeps the original. When the client has already gone there is nothing to report, and the part is dropped.

func (*Response) Streaming

func (r *Response) Streaming() bool

Streaming reports which shape this turn took.

type Result

type Result struct {
	// Streaming reports which shape the turn took.
	Streaming bool
	// Committed reports whether any byte reached the client. It is the line
	// between an error response and a truncated answer.
	Committed bool
	// Disconnected reports that the client went away before the turn ended.
	Disconnected bool
	// Abandoned reports that the handler returned without calling Close. The
	// framework ended the turn before the response was recycled: nothing was
	// written after the handler returned, a part still pending was dropped,
	// and the client got no terminator. Committed says whether anything had
	// reached the client before that.
	Abandoned bool

	// Text is the assembled answer.
	Text string
	// Reasoning is the assembled reasoning trace.
	Reasoning string
	// ToolCalls are the tool calls the model asked for, as the parts that
	// carried them: a tool-call part already holds the id, the name and the raw
	// arguments, so a second type for the same data would only be one more thing
	// to keep in step.
	ToolCalls []Part
	// Usage is what the turn cost, accumulated across every usage part.
	Usage Usage
	// Parts is every part, when Options.KeepParts is set.
	Parts []Part
	// Err is why the turn ended badly, or nil.
	Err error

	// TTFP is the time to the first part, which is the latency a reader
	// actually perceives. Total duration is dominated by how long the answer is.
	TTFP time.Duration
	// Duration is the whole turn.
	Duration time.Duration
}

Result is what a turn produced. It is handed to OnFinish and available from Response.Result.

type Sink

type Sink interface {
	// Frame writes one SSE frame, or returns ErrNotFramed.
	Frame(ev sse.Event) error
	// Write writes raw bytes, or returns ErrNotRaw.
	io.Writer
	// Flush pushes whatever has been written to the client.
	Flush() error
}

Sink is what an Encoder writes into.

An encoder uses whichever half matches the transport it asked for through ContentType: an empty content type gets an event stream and uses Frame, and anything else gets a plain streaming response and uses Write.

type TagSplitter

type TagSplitter struct {
	// contains filtered or unexported fields
}

TagSplitter separates text wrapped in an inline tag from the rest, across chunk boundaries.

Several models emit their reasoning inline, fenced by a tag such as <think>, rather than in a separate field. The tag arrives split across deltas as often as not - "<th" in one chunk and "ink>" in the next - so a naive strings.Contains on each chunk both misses the tag and leaks its characters into the answer.

It holds a small buffer, not a stream: feed it deltas, take the parts it returns. Use it through the pointer TagSplit returns; it is not safe for concurrent use.

func TagSplit

func TagSplit(id, open, close string) *TagSplitter

TagSplit returns a splitter for text fenced by open and close, emitting the fenced text as reasoning and everything else as answer text.

splitter := ai.TagSplit(msgID, "<think>", "</think>")
for _, p := range splitter.Write(delta) {
	res.Send(p)
}
// ... at the end of the stream:
for _, p := range splitter.Flush() {
	res.Send(p)
}

It panics on an empty marker: an empty string is found at every position, so the splitter would loop forever or call the whole answer reasoning, and no model emits an invisible tag.

func (*TagSplitter) Flush

func (s *TagSplitter) Flush() []Part

Flush releases whatever is still held back, at the end of a stream.

A tag that never completed is not a tag: its characters are part of the answer, and dropping them would silently truncate it.

func (*TagSplitter) Inside

func (s *TagSplitter) Inside() bool

Inside reports whether the splitter is currently inside the tag.

func (*TagSplitter) StartInside

func (s *TagSplitter) StartInside() *TagSplitter

StartInside tells the splitter that the stream begins inside the tag.

It exists for models that emit a closing tag without ever having emitted an opening one, which is a real and otherwise baffling dialect: everything before the first </think> is reasoning.

func (*TagSplitter) Write

func (s *TagSplitter) Write(delta string) []Part

Write feeds one delta and returns the parts it completed.

It holds back only what could still turn out to be the start of a tag, so a tag split across any number of chunks is recognised, and at most len(tag)-1 characters are ever delayed.

type UpstreamError

type UpstreamError struct {
	// Status is the HTTP status code, or 0 when the request never got one.
	Status int
	// Class is what to do about it.
	Class Class
	// Message is the provider's own message, when the body carried one in any of
	// the shapes providers use.
	Message string
	// Code is the provider's machine-readable code, when there was one.
	Code string
	// RequestID is the provider's request identifier, from the body or from one
	// of the usual headers. It is the only thing that makes a support
	// conversation about a single failed generation possible.
	RequestID string
	// RetryAfter is the delay the provider asked for, from Retry-After.
	RetryAfter time.Duration
	// Body is the response body, truncated. Nil when there was no response.
	Body []byte
	// contains filtered or unexported fields
}

UpstreamError is a typed failure from a provider.

func AsUpstream

func AsUpstream(err error) (*UpstreamError, bool)

AsUpstream extracts an *UpstreamError from an error chain.

func Classify

func Classify(resp *http.Response, err error, opts ...ClassifyOptions) *UpstreamError

Classify turns an upstream exchange into a typed failure, or nil when nothing went wrong.

It takes the response and not only the error, and that is the whole point. A well-behaved Go HTTP client returns a non-2xx response with a nil error, so a Classify(err) signature would have to recover the status code from a string and would recreate exactly the type erasure it exists to delete.

It reads and closes the response body on failure, keeping up to MaxBody of it, because the body is where the provider put the reason. On success it leaves the body untouched for the caller to stream.

func (*UpstreamError) Error

func (e *UpstreamError) Error() string

Error implements error.

func (*UpstreamError) ErrorCode

func (e *UpstreamError) ErrorCode() errors.ErrorCodeName

ErrorCode returns the canonical rest/errors code for this failure.

The mapping is deliberate, and it is not a passthrough of the upstream's own status. A provider 401 means this gateway's credential is finished, which is not the caller's problem to fix; telling them they are unauthorised sends them to check a key they do not own.

It returns an empty code for a cancelled request, which means there is nothing to report: the caller has gone, so nothing should be written and nobody should be paged. Test it with ErrorCodeName.String() == "", the same sentinel the errors package uses elsewhere.

func (*UpstreamError) HTTPStatus

func (e *UpstreamError) HTTPStatus() int

HTTPStatus is the status code to give the client for this failure.

It is not the upstream's status code, and the difference matters. A 401 from a provider means the gateway's credential is finished, which is a 502 to the caller: telling them they are unauthorised would send them off to check their own key. The one code that does pass through is 429, because rate limiting is the caller's problem too and the delay is worth forwarding.

func (*UpstreamError) Report

func (e *UpstreamError) Report(ctx *context.Context) *errors.Error

Report runs this failure through the rest/errors pipeline under its canonical code, returning the client-safe Error, or nil when there is nothing to report.

The client is told what the failure means, never what the provider said. A provider's message is text this service did not write, about a system the caller cannot see, and it routinely names the vendor, the model or the host; forwarding it is a leak whatever the status code was. It goes to the logger and the collector, where it is useful, and the caller gets a sentence derived from the class.

The error is wrapped as private on the way in, so even if this function's own text were removed the pipeline would still redact rather than forward.

func (*UpstreamError) Retryable

func (e *UpstreamError) Retryable() bool

Retryable reports whether trying again could plausibly help.

func (*UpstreamError) Timeout

func (e *UpstreamError) Timeout() bool

Timeout reports whether the request failed because the upstream did not answer in time: a context deadline, or a transport error whose Timeout() is true. net/http's own timeout errors - http.Client.Timeout, Transport.ResponseHeaderTimeout - are both at once; the net.Error check is for timeouts from elsewhere, such as a net.Conn read deadline, which matches os.ErrDeadlineExceeded and not context.DeadlineExceeded. It is transient - a stalled provider is the failure a gateway sees most - and it is reported to the client as DEADLINE_EXCEEDED.

func (*UpstreamError) Unwrap

func (e *UpstreamError) Unwrap() error

Unwrap exposes the transport error, so errors.Is against context.Canceled and the net package's own errors keeps working.

type Usage

type Usage struct {
	// omitzero, not omitempty: under encoding/json/v2 omitempty no longer
	// omits a zero number.
	InputTokens       int64 `json:"input_tokens,omitzero"`
	OutputTokens      int64 `json:"output_tokens,omitzero"`
	CachedInputTokens int64 `json:"cached_input_tokens,omitzero"`
	ReasoningTokens   int64 `json:"reasoning_tokens,omitzero"`

	// Extra carries counters a provider reports that this struct does not name,
	// so nothing is lost in translation.
	Extra map[string]int64 `json:"extra,omitempty"`
}

Usage is what one turn cost.

Cost in money is deliberately absent: pricing a provider means tracking every provider's price list forever, and it changes more often than this package would ship. Compute it in OnFinish, where you already know which model ran.

func (*Usage) Add

func (u *Usage) Add(u2 Usage)

Add accumulates u2 into u. Providers report usage in pieces - some at the start of a stream, some at the end - and a relay has to add them up.

func (Usage) IsZero

func (u Usage) IsZero() bool

IsZero reports whether anything was counted.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL