Documentation
ΒΆ
Overview ΒΆ
Package genai is the opiniated high performance professional-grade AI package for Go.
It provides a generic interface to interact with various LLM providers, while allowing full access to each provider's full capabilities.
Check out the examples for a quick start.
Example (GenSyncWithToolCallLoop_with_custom_HTTP_Header) ΒΆ
package main
import (
"context"
"fmt"
"log"
"net/http"
"time"
"github.com/maruel/roundtrippers"
"github.com/maruel/genai"
"github.com/maruel/genai/adapters"
"github.com/maruel/genai/providers/anthropic"
)
func main() {
// Modified version of the example in package adapters, with a custom header.
//
// As of June 2025, interleaved thinking can be enabled with a custom header.
// https://docs.anthropic.com/en/docs/build-with-claude/extended-thinking#interleaved-thinking
wrapper := func(h http.RoundTripper) http.RoundTripper {
return &roundtrippers.Header{
Transport: h,
Header: http.Header{"anthropic-beta": []string{"interleaved-thinking-2025-05-14"}},
}
}
ctx := context.Background()
c, err := anthropic.New(ctx, genai.ProviderOptionModel("claude-sonnet-4-20250514"), genai.ProviderOptionTransportWrapper(wrapper))
if c != nil {
defer func() {
if err := c.Close(); err != nil {
log.Printf("Close: %v", err)
}
}()
}
if err != nil {
log.Print(err)
return
}
msgs := genai.Messages{genai.NewTextMessage("What season is MontrΓ©al currently in?")}
opts := genai.GenOptionTools{
Tools: []genai.ToolDef{locationClockTime},
// Force the LLM to do a tool call first.
Force: genai.ToolCallRequired,
}
newMsgs, _, err := adapters.GenSyncWithToolCallLoop(ctx, c, msgs, &opts)
if err != nil {
log.Print(err)
return
}
fmt.Printf("%s\n", newMsgs[len(newMsgs)-1].String())
}
var locationClockTime = genai.ToolDef{
Name: "get_today_date_current_clock_time",
Description: "Get the current clock time and today's date.",
Callback: func(ctx context.Context, e *location) (string, error) {
if e.Location != "MontrΓ©al" {
return "ask again with MontrΓ©al", nil
}
return time.Now().Format("Monday 2006-01-02 15:04:05"), nil
},
}
type location struct {
Location string `json:"location" json_description:"Location to ask the current time in"`
}
Output:
Index ΒΆ
- Constants
- type Answer
- type Answers
- type Array
- type AudioFormat
- type CacheEntry
- type Citation
- type CitationSource
- type CitationType
- type DecisionContent
- type DecisionUsage
- type Doc
- type EffortLevel
- type EmbeddingRequest
- type EmbeddingResponse
- type FinishReason
- type GenOption
- type GenOptionAudio
- type GenOptionImage
- type GenOptionPollInterval
- type GenOptionSeed
- type GenOptionText
- type GenOptionTools
- type GenOptionVideo
- type GenOptionWeb
- type JSONSchema
- type Job
- type Logprob
- type Message
- func (m *Message) Accumulate(mf *Reply) error
- func (m *Message) Decode(x any) error
- func (m *Message) DoToolCalls(ctx context.Context, tools []ToolDef) (Message, error)
- func (m *Message) GoString() string
- func (m *Message) IsZero() bool
- func (m *Message) Reasoning() string
- func (m *Message) Role() string
- func (m *Message) String() string
- func (m *Message) UnmarshalJSON(b []byte) error
- func (m *Message) Validate() error
- type Messages
- type Modalities
- type Modality
- type Model
- type NoulCriteria
- type Object
- type Provider
- type ProviderCapabilities
- type ProviderOption
- type ProviderOptionAPIKey
- type ProviderOptionModalities
- type ProviderOptionModel
- type ProviderOptionPreloadedModels
- type ProviderOptionRemote
- type ProviderOptionStarterWrapper
- type ProviderOptionTransportWrapper
- type ProviderPing
- type ProviderScoreboardVariants
- type ProviderUnwrap
- type Question
- type QuestionType
- type Questions
- type RateLimit
- type RateLimitPeriod
- type RateLimitType
- type RealtimeCancel
- type RealtimeCapabilities
- type RealtimeContent
- type RealtimeError
- type RealtimeEvent
- type RealtimeEventKind
- type RealtimeInput
- type RealtimeInputContent
- type RealtimeOption
- type RealtimeOptionAudio
- type RealtimeOptionInstructions
- type RealtimeOptionTools
- type RealtimeProviderEvent
- type RealtimeSession
- type RealtimeSessionInfo
- type RealtimeToolCallCancelledError
- type RealtimeTruncate
- type RealtimeTurnDetection
- type RealtimeUpdate
- type RealtimeUpdateCapabilities
- type ReasoningOutput
- type Reply
- type Request
- type Result
- type ScoreLegend
- type ScoreboardVariant
- type Starter
- type SystemOneRequest
- type SystemOneResponse
- type Text
- type ToolCall
- type ToolCallRequest
- type ToolCallResult
- type ToolDef
- type Usage
Examples ΒΆ
Constants ΒΆ
const ( // ModalityAudio is support for audio formats like MP3, WAV, Opus, Flac, etc. ModalityAudio = scoreboard.ModalityAudio // ModalityDecision is typed answers returned by SystemOne. ModalityDecision = scoreboard.ModalityDecision // ModalityDocument is support for PDF with multi-modal comprehension, both images and text. This includes // code blocks. ModalityDocument = scoreboard.ModalityDocument // ModalityEmbedding is a vector returned by Embed. ModalityEmbedding = scoreboard.ModalityEmbedding // ModalityImage is support for image formats like PNG, JPEG, often single frame GIF, and WEBP. ModalityImage = scoreboard.ModalityImage // ModalityText is for raw text. ModalityText = scoreboard.ModalityText // ModalityVideo is support for video formats like MP4 or MKV. ModalityVideo = scoreboard.ModalityVideo )
Variables ΒΆ
This section is empty.
Functions ΒΆ
This section is empty.
Types ΒΆ
type Answer ΒΆ added in v0.10.0
type Answer struct {
// Type is the type of the question this is the answer to.
Type QuestionType `json:"type"`
// Noul is the probability that the answer is yes, from 0 to 1. It is set for QuestionNoul.
Noul float64 `json:"noul,omitzero"`
// Choice is the highest probability option. It is set for QuestionChoice.
Choice string `json:"choice,omitzero"`
// Score is the probability weighted value across the levels. It can land between levels. It is set for
// QuestionScore.
Score float64 `json:"score,omitzero"`
// Confidence is how certain the model is, derived from Probabilities. It is set for QuestionChoice and
// QuestionScore.
Confidence float64 `json:"confidence,omitzero"`
// Probabilities maps each option, or each level as a string key, to its probability. It is set for
// QuestionChoice and QuestionScore.
Probabilities map[string]float64 `json:"probabilities,omitzero"`
// Legend maps each level as a string key to the description passed in Question.Score for that level.
// It is set for QuestionScore.
Legend ScoreLegend `json:"legend,omitzero"`
// contains filtered or unexported fields
}
Answer is the answer to a Question.
It is a union discriminated by Type. Only the fields that match Type are set, and MarshalJSON writes only those fields, with the shape the API uses, so a probability or a score of 0 is not dropped.
An answer type the server adds later is kept as is instead of making the whole reply unusable.
func (*Answer) MarshalJSON ΒΆ added in v0.10.0
MarshalJSON implements json.Marshaler.
The fields of the union that do not match Type are omitted, even when they are zero, so that a score of 0 or a probability of 0 is preserved.
func (*Answer) UnmarshalJSON ΒΆ added in v0.10.0
UnmarshalJSON implements json.Unmarshaler.
type Answers ΒΆ added in v0.10.0
Answers holds the answer to each question, keyed by the question name. Entries must not be nil.
type Array ΒΆ added in v0.10.0
type Array []any
Array is DecisionContent that is a JSON array.
The values are any, like Object: they must be JSON encodable. They are not restricted to DecisionContent items.
type AudioFormat ΒΆ added in v0.11.0
type AudioFormat struct {
SampleRate int
}
AudioFormat describes raw mono PCM16LE at SampleRate Hz. genai never resamples; callers convert at the capture and playback boundaries.
func (*AudioFormat) Validate ΒΆ added in v0.11.0
func (f *AudioFormat) Validate() error
Validate checks that SampleRate is positive.
type CacheEntry ΒΆ
CacheEntry is one file (or GenSync request) cached on the provider for reuse.
type Citation ΒΆ
type Citation struct {
// CitedText is the text that was cited.
CitedText string `json:"cited_text,omitzero"`
// StartIndex is the starting character position of the citation in the answer (0-based).
StartIndex int64 `json:"start_index,omitzero"`
// EndIndex is the ending character position of the citation in the answer (0-based, exclusive).
EndIndex int64 `json:"end_index,omitzero"`
// Sources contains information about the source documents or tools that support this citation.
Sources []CitationSource `json:"sources,omitzero"`
// contains filtered or unexported fields
}
Citation represents a reference to source material that supports content. It provides a unified interface for different provider citation formats.
Normally one of CitedText or StartIndex/EndIndex is set.
type CitationSource ΒΆ
type CitationSource struct {
// Type indicates the source type.
Type CitationType `json:"type,omitzero"`
// ID is a unique identifier for the source (e.g., document ID, tool call ID).
ID string `json:"id,omitzero"`
// Title is the human-readable title of the source.
Title string `json:"title,omitzero"`
// URL is the web URL for the source, if applicable.
URL string `json:"url,omitzero"`
// Snippet is a snippet from the source, if applicable. It is the web search query for CitationWebQuery.
Snippet string `json:"snippet,omitzero"`
// StartCharIndex is the starting character position of the citation in the sourced document (0-based).
StartCharIndex int64 `json:"start_index,omitzero"`
// EndCharIndex is the ending character position of the citation in the sourced document (0-based, exclusive).
EndCharIndex int64 `json:"end_index,omitzero"`
// StartPageNumber is the starting page number of the citation in the sourced document (1-based).
StartPageNumber int64 `json:"start_page_number,omitzero"`
EndPageNumber int64 `json:"end_page_number,omitzero"`
// StartBlockIndex is the starting block index of the citation in the sourced document (0-based).
StartBlockIndex int64 `json:"start_block_index,omitzero"`
EndBlockIndex int64 `json:"end_block_index,omitzero"`
// Date is the date of the source, if applicable.
Date string `json:"date,omitzero"`
// Metadata contains additional source-specific information.
// For document sources: document index, page numbers, etc.
// For tool sources: tool output, function name, etc.
// For web sources: encrypted index, search result info, etc.
Metadata map[string]any `json:"metadata,omitzero"`
// contains filtered or unexported fields
}
CitationSource represents a source that supports a citation.
func (*CitationSource) IsZero ΒΆ
func (cs *CitationSource) IsZero() bool
IsZero returns true if the citation source is empty.
func (*CitationSource) Validate ΒΆ
func (cs *CitationSource) Validate() error
Validate ensures the citation source is valid.
type CitationType ΒΆ
type CitationType int32
CitationType is a citation that a model returned as part of its reply.
const ( // CitationWebQuery is an query used as part of a web search. CitationWebQuery CitationType = iota + 1 // CitationWeb is an URL from a web search. CitationWeb // CitationWebImage is an URL to an image from a web search. CitationWebImage // CitationDocument is from a document provided as input or explicitly referenced. CitationDocument // CitationTool is when the provider refers to the result of a tool call in its answer. CitationTool )
type DecisionContent ΒΆ added in v0.10.0
type DecisionContent interface {
// Validate ensures the content is valid.
Validate() error
// contains filtered or unexported methods
}
DecisionContent is text, a JSON object, or a JSON array.
It is implemented by Text, Object and Array, and by JSON state read from a Doc. It marshals to the value itself, not to an object with fields. Use nil when the field is optional and is left unset.
It is used for state, instructions and criteria descriptions. It is meant to be sent; the API only returns it in Answer.Legend, decoded back by ScoreLegend, since an interface cannot be decoded on its own.
func ParseDecisionContent ΒΆ added in v0.10.0
func ParseDecisionContent(b []byte) (DecisionContent, error)
ParseDecisionContent decodes one JSON value into its DecisionContent variant.
The variant is picked from the JSON token, not from the Go value a decoder would produce for an any, so what it accepts is unambiguous.
type DecisionUsage ΒΆ added in v0.10.0
type DecisionUsage struct {
// InputTokens counts the prompt tokens evaluated for all questions.
InputTokens int64 `json:"input_tokens"`
// OutputTokens counts output tokens reported by the provider, often zero for decision models.
OutputTokens int64 `json:"output_tokens"`
// ReasoningTokens counts reasoning tokens within OutputTokens, when reported by the provider.
ReasoningTokens int64 `json:"reasoning_tokens,omitzero"`
}
DecisionUsage reports the token usage of a request.
func (*DecisionUsage) Validate ΒΆ added in v0.10.0
func (u *DecisionUsage) Validate() error
Validate rejects negative token counts.
type Doc ΒΆ
type Doc struct {
// Filename is the name of the file. For many providers, only the extension
// is relevant. They only use mime-type, which is derived from the filename's
// extension. When an URL is provided or when the object provided to Document
// implements a method with the signature `Name() string`, like an
// `*os.File`, Filename is optional.
Filename string `json:"filename,omitzero"`
// Src is raw document data. It is perfectly fine to use a bytes.NewReader() or *os.File.
Src io.ReadSeeker `json:"bytes,omitzero"`
// URL is the reference to the raw data. When set, the mime-type is derived from the URL.
URL string `json:"url,omitzero"`
// contains filtered or unexported fields
}
Doc is a document.
func (*Doc) GetFilename ΒΆ
GetFilename returns the filename to use for the document, querying the Document's name if available.
func (*Doc) MIMEType ΒΆ added in v0.11.0
MIMEType returns the mime type the filename or URL declares, or an empty string when the extension is unknown.
func (*Doc) MarshalJSON ΒΆ
MarshalJSON implements the json.Marshaler interface.
func (*Doc) Read ΒΆ
Read reads the document content into memory.
maxSize must be positive. Oversized content is rejected, including unseekable sources and content that grows after its size is checked. It returns the mime type, the raw bytes and an error if any.
func (*Doc) UnmarshalJSON ΒΆ
UnmarshalJSON implements the json.Unmarshaler interface.
type EffortLevel ΒΆ added in v0.11.0
type EffortLevel string
EffortLevel controls how much effort a model spends reasoning about its response. Providers map these levels to their native controls; the token budget and latency of a level depend on the provider and model. Empty preserves the provider's default.
const ( // EffortOff disables thinking where possible, or selects the lowest supported effort. EffortOff EffortLevel = "off" // EffortMinimal requests the smallest supported reasoning effort without disabling thinking. // Providers use native minimal effort where supported, or their lowest enabled level. EffortMinimal EffortLevel = "minimal" // EffortLow favors lower cost and latency. EffortLow EffortLevel = "low" // EffortMedium balances reasoning quality, cost, and latency. EffortMedium EffortLevel = "medium" // EffortHigh favors reasoning quality over cost and latency. EffortHigh EffortLevel = "high" // EffortXHigh requests extra-high effort, or the highest supported effort. EffortXHigh EffortLevel = "xhigh" // EffortMax selects the highest supported reasoning effort. EffortMax EffortLevel = "max" )
func (EffortLevel) Validate ΒΆ added in v0.11.0
func (l EffortLevel) Validate() error
Validate ensures the effort level is valid.
type EmbeddingRequest ΒΆ added in v0.10.2
type EmbeddingRequest struct {
// Inputs contains one text or document block per output vector.
Inputs []Request `json:"inputs"`
// Dimensions is the requested vector size, or zero for the model default.
// Support depends on the provider and model.
Dimensions int `json:"dimensions,omitzero"`
}
EmbeddingRequest embeds a batch of content with the client's configured model. Include any task prefixes required by that model in each request's Text.
func (*EmbeddingRequest) Validate ΒΆ added in v0.10.2
func (r *EmbeddingRequest) Validate() error
Validate implements Validatable.
type EmbeddingResponse ΒΆ added in v0.10.2
type EmbeddingResponse struct {
Embeddings [][]float32 `json:"embeddings"`
// Usage is zero when the provider doesn't report token counts.
Usage Usage `json:"usage,omitzero"`
}
EmbeddingResponse contains one vector per input request, in input order. Values are preserved as returned by the provider. Normalization depends on the provider and model; callers comparing vectors must account for that contract.
func (*EmbeddingResponse) Validate ΒΆ added in v0.10.2
func (r *EmbeddingResponse) Validate() error
Validate implements Validatable. It checks uniform dimensions and finite, nonzero vectors without normalizing their values.
type FinishReason ΒΆ
type FinishReason string
FinishReason is the reason why the model stopped generating tokens.
It can be one of the well known below or a custom value.
const ( // FinishedStop means the LLM was done for the turn. Some providers confuse it with // FinishedStopSequence. FinishedStop FinishReason = "stop" // FinishedLength means the model reached the maximum number of tokens allowed as set in // GenOptionText.MaxTokens or as limited by the provider. FinishedLength FinishReason = "length" // FinishedToolCalls means the model called one or multiple tools and needs the replies to continue the turn. FinishedToolCalls FinishReason = "tool_calls" // FinishedStopSequence means the model stopped because it saw a stop word as listed in GenOptionText.Stop. FinishedStopSequence FinishReason = "stop" // FinishedContentFilter means the model stopped because the reply got caught by a content filter. FinishedContentFilter FinishReason = "content_filter" // Pending means that it's not finished yet. For use with ProviderGenAsync. Pending FinishReason = "pending" )
type GenOption ΒΆ added in v0.2.0
type GenOption interface {
// Validate ensures the options object is valid.
Validate() error
}
GenOption is options that can be provided to Provider.GenSync and Provider.GenStream.
type GenOptionAudio ΒΆ added in v0.2.0
type GenOptionAudio struct {
// contains filtered or unexported fields
}
GenOptionAudio specifies audio generation options.
func (*GenOptionAudio) Validate ΒΆ added in v0.2.0
func (o *GenOptionAudio) Validate() error
Validate implements Validatable.
type GenOptionImage ΒΆ added in v0.2.0
GenOptionImage is a list of frequent options supported by most ProviderDoc. Each provider is free to support more options through a specialized struct.
func (*GenOptionImage) Validate ΒΆ added in v0.2.0
func (o *GenOptionImage) Validate() error
Validate ensures the completion options are valid.
type GenOptionPollInterval ΒΆ
GenOptionPollInterval is the time interval to poll generation progress when using GenSync.
func (GenOptionPollInterval) Validate ΒΆ
func (p GenOptionPollInterval) Validate() error
Validate ensures the poll interval is valid.
type GenOptionSeed ΒΆ added in v0.2.0
type GenOptionSeed int64
GenOptionSeed is the seed for the random number generator.
func (GenOptionSeed) Validate ΒΆ added in v0.2.0
func (s GenOptionSeed) Validate() error
Validate ensures the seed is valid.
type GenOptionText ΒΆ added in v0.2.0
type GenOptionText struct {
// Temperature adjust the creativity of the sampling. Generally between 0 and 2.
Temperature float64
// TopP adjusts correctness sampling between 0 and 1. The higher the more diverse the output.
TopP float64
// MaxTokens is the maximum number of tokens to generate. Used to limit it
// lower than the default maximum, for budget reasons.
MaxTokens int64
// TopLogprobs requests to return the top logprobs in the reply.
TopLogprobs int64
// SystemPrompt is the prompt to use for the system role.
SystemPrompt string
// TopK adjusts sampling where only the N first candidates are considered.
TopK int64
// Stop is the list of tokens to stop generation.
Stop []string
// ReplyAsJSON enforces the output to be valid JSON, any JSON. It is
// important to tell the model to reply in JSON in the prompt itself.
ReplyAsJSON bool
// DecodeAs enforces a reply with a specific JSON structure. It must be either a JSON object or array
// that can be decoded by encoding/json and can have jsonschema tags, usually a pointer to a struct, or
// a JSONSchema.
//
// Not every provider accepts an array at the root of the reply, e.g. OpenAI and Anthropic require an
// object. Wrap the slice in a struct with a single field when the provider rejects it.
//
// It is important to request the model to "reply in JSON" in the prompt itself.
//
// If you use a JSONSchema, it will be used to validate the reply.
//
// If you use a pointer to a struct, it is recommended to use jsonschema_description tags to describe each
// field or argument. Use jsonschema:"enum=..." to enforce a specific value within a set. Use omitempty to
// make the field optional. See https://github.com/invopop/jsonschema#example for more examples.
DecodeAs any
// contains filtered or unexported fields
}
GenOptionText is a list of frequent options supported by most Provider with text output modality. Each provider is free to support more options through a specialized struct.
The first group are options supported by (nearly) all providers.
The second group are options supported only by some providers. Using them may cause the chat operation to return a base.ErrNotsupported.
The third group are options supported by a few providers and a few models on each, that will slow down generation (increase latency) and will increase token use (cost).
func (*GenOptionText) DecodeSchema ΒΆ added in v0.6.1
func (o *GenOptionText) DecodeSchema() (JSONSchema, error)
DecodeSchema returns the JSONSchema for the DecodeAs field.
If DecodeAs is a JSONSchema instance, it is returned directly. Otherwise it uses invopop/jsonschema to construct the schema from the type's fields and tags.
func (*GenOptionText) Validate ΒΆ added in v0.2.0
func (o *GenOptionText) Validate() error
Validate ensures the completion options are valid.
type GenOptionTools ΒΆ added in v0.2.0
type GenOptionTools struct {
// Tools is the list of tools that the LLM can request to call.
Tools []ToolDef
// Force tells the LLM a tool call must be done, or not.
Force ToolCallRequest
}
GenOptionTools specifies tool calling options.
func (*GenOptionTools) Validate ΒΆ added in v0.2.0
func (o *GenOptionTools) Validate() error
Validate ensures the completion options are valid.
type GenOptionVideo ΒΆ added in v0.2.0
type GenOptionVideo struct {
// Duration of the video to generate, if supported.
//
// Veo 2 supports only between 5 and 8 seconds and Veo 3 only supports 8 seconds.
Duration time.Duration
// contains filtered or unexported fields
}
GenOptionVideo specifies video generation options.
func (*GenOptionVideo) Validate ΒΆ added in v0.2.0
func (o *GenOptionVideo) Validate() error
Validate implements Validatable.
type GenOptionWeb ΒΆ added in v0.2.0
type GenOptionWeb struct {
// Search specifies if websearch should be enabled. It is generally disabled by default.
//
// # Warning
//
// This will become a structure to provide information about included and excluded domains, and the user's
// location.
Search bool
// Fetch specifies if web fetch should be enabled. When enabled, the LLM can fetch content from URLs.
//
// Currently supported by Anthropic and Gemini.
Fetch bool
}
GenOptionWeb specifies web access options.
func (*GenOptionWeb) Validate ΒΆ added in v0.2.0
func (o *GenOptionWeb) Validate() error
Validate implements GenOption.
type JSONSchema ΒΆ added in v0.6.1
type JSONSchema = json.RawMessage
JSONSchema is a JSON Schema document encoded as raw JSON bytes.
type Logprob ΒΆ
type Logprob struct {
ID int64 `json:"id,omitzero"` // Input token ID.
Text string `json:"text,omitzero"` // Text in UTF-8.
Logprob float64 `json:"logprob"` // Log probability of the token. It should normally be non-zero but sometimes it is.
}
Logprob represents a single log probability information for a token.
One of ID or Text must be set.
type Message ΒΆ
type Message struct {
// Requests is the content from the user.
//
// It is normally a single message. It is more frequently multiple items when using multi-modal content.
Requests []Request `json:"request,omitzero"`
// User must only be used when sent by the user. Only some provider (e.g. OpenAI, Groq, DeepSeek) support it.
User string `json:"user,omitzero"`
// Replies is the message from the LLM.
//
// It is generally a single reply with text or a tool call. Some models can emit multiple content blocks,
// either multi modal or multiple text blocks: a code block and a different block with an explanantion. Some
// models can emit multiple tool calls at once.
Replies []Reply `json:"reply,omitzero"`
// ToolCallResult is the result for a tool call that the LLM requested to make.
//
// These messages are generated by the "computer".
ToolCallResults []ToolCallResult `json:"tool_call_results,omitzero"`
// contains filtered or unexported fields
}
Message is a part of an exchange with a LLM.
It is effectively a union, with the exception of the User field that can be set with In.
func NewTextMessage ΒΆ
NewTextMessage is a shorthand function to create a Message with a single text block.
func (*Message) Accumulate ΒΆ
Accumulate adds a Reply to the message being streamed.
It is used by GenStream. There's generally no need to call it by end users.
func (*Message) Decode ΒΆ
Decode decodes the JSON message into the struct.
For generation replies, request JSON with ReplyAsJSON or DecodeAs in GenOptionText.
Note: this doesn't verify the type is the same as specified in GenOptionText.DecodeAs.
func (*Message) DoToolCalls ΒΆ
DoToolCalls processes all the ToolCall in the Reply if any.
Returns a Message to be added back to the list of messages, only if msg.IsZero() is true.
func (*Message) GoString ΒΆ
GoString returns a JSON representation of the reply for debugging purposes.
func (*Message) String ΒΆ
String is a short hand to get the request or reply content as text.
It ignores reasoning or multi-modal content.
func (*Message) UnmarshalJSON ΒΆ
UnmarshalJSON adds validation during decoding.
type Messages ΒΆ
type Messages []Message
Messages is a chronological list of messages in an exchange with a LLM.
Adjacent messages may have the same role. Tool calls may remain unresolved, and user messages and other tool results may intervene before their results arrive. Providers may impose additional constraints. Conversion must preserve event order without merging messages or inventing tool results.
type Modalities ΒΆ
type Modalities []scoreboard.Modality
Modalities represents the modality supported by the provider in a specific scenario. It can be multiple modalities in multi-modals scenarios.
func (Modalities) String ΒΆ
func (m Modalities) String() string
type Modality ΒΆ
type Modality = scoreboard.Modality
Modality is a modality supported by the provider.
It is aliased from scoreboard.Modality to avoid a circular dependency.
type Model ΒΆ
type Model interface {
GetID() string
String() string
// Context returns the number of tokens the model can process as input.
Context() int64
}
Model represents a served model by the provider.
Use Provider.ListModels() to get a list of models.
type NoulCriteria ΒΆ added in v0.10.0
type NoulCriteria struct {
// True describes what a yes (value near 1) means.
True DecisionContent `json:"true,omitzero"`
// False describes what a no (value near 0) means.
False DecisionContent `json:"false,omitzero"`
}
NoulCriteria describes what the yes and the no answers of a noul question mean.
func (*NoulCriteria) Validate ΒΆ added in v0.10.0
func (c *NoulCriteria) Validate() error
Validate ensures the criteria are valid.
type Object ΒΆ added in v0.10.0
Object is DecisionContent that is a JSON object.
The values are any, like Reply.Opaque: they must be JSON encodable. Go has no recursive JSON value type that does not require wrapping every scalar, so a nested map[string]any, []any or a struct with JSON tags is passed as is. It is the equivalent of the SDKs' Mapping[str, JSONValue | None].
type Provider ΒΆ
type Provider interface {
// Close releases resources held by the provider.
io.Closer
// Name returns the name of the provider.
Name() string
// ModelID returns the model currently used by the provider. It can be an empty string.
ModelID() string
// OutputModalities returns the output modalities supported by this specific client configuration.
//
// This states what kind of output the model will generate (text, audio, image, video, decision). It varies per
// provider and models. The vast majority of providers and models support only output modality like
// text-only, image-only, etc.
OutputModalities() Modalities
// Capabilities returns the optional capabilities this provider supports.
Capabilities() ProviderCapabilities
// Scoreboard returns what the provider supports.
//
// Some models have more features than others, e.g. some models may be text-only while others have vision or
// audio support.
//
// The client code may be the limiting factor for some models, and not the provider itself.
//
// Capability results come from smoke tests; voice catalogs describe
// provider-advertised choices.
Scoreboard() scoreboard.Score
// HTTPClient returns the underlying http client. It may be necessary to use it to fetch the results from
// the provider. An example is retrieving Veo 3 generated videos from Gemini requires the authentication
// headers to be set.
HTTPClient() *http.Client
// GenSync runs generation synchronously.
//
// Multiple options can be mixed together, both standard ones like *GenOptionImage, *GenOptionText,
// *GenOptionTools and provider-specialized options struct, e.g. *anthropic.GenOptionText, *gemini.GenOption.
GenSync(ctx context.Context, msgs Messages, opts ...GenOption) (Result, error)
// GenStream runs generation synchronously, yielding the fragments of replies as the server sends them.
//
// No need to accumulate the fragments into a Message since the Result contains the accumulated message.
GenStream(ctx context.Context, msgs Messages, opts ...GenOption) (iter.Seq[Reply], func() (Result, error))
// SystemOne answers typed questions about a state without text generation or streaming.
// It uses the client's configured model. The request is not modified.
// Requires ProviderCapabilities.SystemOne. Returns base.ErrNotSupported otherwise.
SystemOne(ctx context.Context, req *SystemOneRequest) (*SystemOneResponse, error)
// Embed returns one vector per input request in order using the configured model.
// Configure an explicit embedding model; automatic generation model selection
// does not choose one. Content blocks and options are not modified; document
// readers may be read and sought. Input size limits depend
// on the provider and model. Returns base.ErrNotSupported when unsupported.
// Provider-specific raw APIs expose additional embedding controls.
// Capabilities().Embed describes endpoint implementation; the scoreboard
// Embedding lookup describes tested support for the configured model.
Embed(ctx context.Context, in *EmbeddingRequest) (*EmbeddingResponse, error)
// ConnectRealtime opens a bidirectional conversation and waits for setup acceptance.
// Requires Capabilities().Realtime. Unsupported session features return base.ErrNotSupported.
ConnectRealtime(ctx context.Context, opts ...RealtimeOption) (RealtimeSession, error)
// ListModels returns the list of models the provider supports. Not all providers support it, some will
// return an ErrorNotSupported. For local providers like llamacpp and ollama, they may return only the
// model currently loaded.
ListModels(ctx context.Context) ([]Model, error)
// GenAsync requests a generation and returns a pending job that can be polled.
//
// Requires ProviderCapabilities.GenAsync to be set. Returns base.ErrNotSupported otherwise.
GenAsync(ctx context.Context, msgs Messages, opts ...GenOption) (Job, error)
// PokeResult requests the state of the job.
//
// When the job is still pending, Result.Usage.FinishReason is Pending.
//
// Requires ProviderCapabilities.GenAsync to be set. Returns base.ErrNotSupported otherwise.
PokeResult(ctx context.Context, job Job) (Result, error)
// CacheAddRequest caches a request.
//
// Requires ProviderCapabilities.Caching to be set. Returns base.ErrNotSupported otherwise.
//
// # Warning
//
// May be changed in the future.
CacheAddRequest(ctx context.Context, msgs Messages, name, displayName string, ttl time.Duration, opts ...GenOption) (string, error)
// CacheList lists the caches entries.
//
// Requires ProviderCapabilities.Caching to be set. Returns base.ErrNotSupported otherwise.
//
// # Warning
//
// May be changed in the future.
CacheList(ctx context.Context) ([]CacheEntry, error)
// CacheDelete deletes a cache entry.
//
// Requires ProviderCapabilities.Caching to be set. Returns base.ErrNotSupported otherwise.
//
// # Warning
//
// May be changed in the future.
CacheDelete(ctx context.Context, name string) error
}
Provider is the base interface that all provider interfaces embed.
The first group contains local methods. Calling these methods will not make an HTTP request.
The second group is supported by the majority of providers.
The rest is supported by a limited number of providers.
Example (Embed) ΒΆ
package main
import (
"context"
"fmt"
"log"
"github.com/maruel/genai"
"github.com/maruel/genai/providers/gemini"
)
func main() {
ctx := context.Background()
c, err := gemini.New(ctx, genai.ProviderOptionModel("gemini-embedding-2"))
if c != nil {
defer func() {
if err := c.Close(); err != nil {
log.Printf("Close: %v", err)
}
}()
}
if err != nil {
log.Fatal(err)
}
if !c.Capabilities().Embed {
log.Fatal("embedding endpoint is unavailable")
}
sb := c.Scoreboard()
if f := sb.Embedding(c.ModelID()); f != nil {
fmt.Printf("Qualified default size: %d\n", f.Dimensions)
} else {
fmt.Println("Embedding model support is unmeasured")
}
out, err := c.Embed(ctx, &genai.EmbeddingRequest{
Inputs: []genai.Request{{Text: "A kitten plays with yarn."}, {Text: "A cat plays with string."}},
Dimensions: 768,
})
if err != nil {
log.Fatal(err)
}
fmt.Printf("Returned %d vectors with %d dimensions\n", len(out.Embeddings), len(out.Embeddings[0]))
}
Output:
Example (GenSync_audio) ΒΆ
package main
import (
"context"
"fmt"
"log"
"os"
"strings"
"github.com/maruel/genai"
"github.com/maruel/genai/providers/gemini"
)
func main() {
// Supported by Gemini, OpenAI.
// Using a free small model for testing.
// See https://ai.google.dev/gemini-api/docs/models/gemini?hl=en
ctx := context.Background()
c, err := gemini.New(ctx, genai.ProviderOptionModel("gemini-2.5-flash-lite"))
if c != nil {
defer func() {
if err := c.Close(); err != nil {
log.Printf("Close: %v", err)
}
}()
}
if err != nil {
log.Print(err)
return
}
f, err := os.Open("internal/internaltest/testdata/mystery_word.mp3")
if err != nil {
log.Print(err)
return
}
defer func() { _ = f.Close() }()
msgs := genai.Messages{
{
Requests: []genai.Request{
{Text: "What is the word said? Reply with only the word."},
{Doc: genai.Doc{Src: f}},
},
},
}
resp, err := c.GenSync(ctx, msgs)
if err != nil {
log.Print(err)
return
}
fmt.Printf("Heard: %v\n", strings.TrimRight(strings.ToLower(resp.String()), "."))
// This would Output: Heard: orange
}
Output:
Example (GenSync_pdf) ΒΆ
package main
import (
"context"
"fmt"
"log"
"os"
"strings"
"github.com/maruel/genai"
"github.com/maruel/genai/providers/gemini"
)
func main() {
// Supported by Anthropic, Gemini, Mistral, OpenAI.
// Using a free small model for testing.
// See https://ai.google.dev/gemini-api/docs/models/gemini?hl=en
ctx := context.Background()
c, err := gemini.New(ctx, genai.ProviderOptionModel("gemini-2.5-flash-lite"))
if c != nil {
defer func() {
if err := c.Close(); err != nil {
log.Printf("Close: %v", err)
}
}()
}
if err != nil {
log.Print(err)
return
}
f, err := os.Open("internal/internaltest/testdata/hidden_word.pdf")
if err != nil {
log.Print(err)
return
}
defer func() { _ = f.Close() }()
msgs := genai.Messages{
{
Requests: []genai.Request{
{Text: "What is the word? Reply with only the word."},
{Doc: genai.Doc{Src: f}},
},
},
}
resp, err := c.GenSync(ctx, msgs)
if err != nil {
log.Print(err)
return
}
fmt.Printf("Hidden word in PDF: %v\n", strings.ToLower(resp.String()))
// This would Output: Hidden word in PDF: orange
}
Output:
Example (GenSync_video) ΒΆ
package main
import (
"context"
"fmt"
"log"
"os"
"strings"
"github.com/maruel/genai"
"github.com/maruel/genai/providers/gemini"
)
func main() {
// Supported by Gemini, TogetherAI.
// Using a free small model for testing.
// See https://ai.google.dev/gemini-api/docs/models/gemini?hl=en
ctx := context.Background()
c, err := gemini.New(ctx, genai.ProviderOptionModel("gemini-2.5-flash"))
if c != nil {
defer func() {
if err := c.Close(); err != nil {
log.Printf("Close: %v", err)
}
}()
}
if err != nil {
log.Print(err)
return
}
f, err := os.Open("internal/internaltest/testdata/animation.mp4")
if err != nil {
log.Print(err)
return
}
defer func() { _ = f.Close() }()
// TogetherAI seems to require separate messages for text and images.
msgs := genai.Messages{
genai.NewTextMessage("What is the word? Reply with exactly and only one word."),
{Requests: []genai.Request{{Doc: genai.Doc{Src: f}}}},
}
resp, err := c.GenSync(ctx, msgs)
if err != nil {
log.Print(err)
return
}
fmt.Printf("Saw: %v\n", strings.ToLower(resp.String()))
// This would Output: Saw: banana
}
Output:
Example (GenSync_vision) ΒΆ
package main
import (
"bytes"
"context"
"fmt"
"log"
"os"
"github.com/maruel/genai"
"github.com/maruel/genai/providers/gemini"
)
func main() {
// Supported by Anthropic, Gemini, Groq, Mistral, Ollama, OpenAI, TogetherAI.
// Using a free small model for testing.
// See https://ai.google.dev/gemini-api/docs/models/gemini?hl=en
ctx := context.Background()
c, err := gemini.New(ctx, genai.ProviderOptionModel("gemini-2.5-flash-lite"))
if c != nil {
defer func() {
if err := c.Close(); err != nil {
log.Printf("Close: %v", err)
}
}()
}
if err != nil {
log.Print(err)
return
}
bananaJpg, err := os.ReadFile("internal/internaltest/testdata/banana.jpg")
if err != nil {
log.Print(err)
return
}
msgs := genai.Messages{
{
Requests: []genai.Request{
{Text: "Is it a banana? Reply with only the word yes or no."},
{Doc: genai.Doc{Filename: "banana.jpg", Src: bytes.NewReader(bananaJpg)}},
},
},
}
resp, err := c.GenSync(ctx, msgs)
if err != nil {
log.Print(err)
return
}
fmt.Printf("Banana: %v\n", resp.String())
// This would Output: Banana: yes
}
Output:
type ProviderCapabilities ΒΆ
type ProviderCapabilities struct {
// Realtime indicates that the client implements bidirectional realtime sessions,
// independent of its configured model. Session capabilities are negotiated by
// ConnectRealtime and reported through RealtimeSession.Info.
Realtime bool
// Embed indicates that the client implements the embedding endpoint, independent
// of its configured model. scoreboard.Score.Embedding reports qualified support;
// a missing measurement means unknown, not unsupported.
Embed bool
// SystemOne indicates the provider implements typed decision inference.
SystemOne bool
// GenAsync indicates the provider supports GenAsync and PokeResult for batch operations.
GenAsync bool
// Caching indicates the provider supports CacheAddRequest, CacheList, and CacheDelete.
Caching bool
// contains filtered or unexported fields
}
ProviderCapabilities describes optional capabilities implemented by a client.
type ProviderOption ΒΆ
type ProviderOption interface {
Validate() error
}
ProviderOption is an option for provider constructors.
type ProviderOptionAPIKey ΒΆ
type ProviderOptionAPIKey string
ProviderOptionAPIKey provides an API key to authenticate to the server.
Most providers require an API key, and the client will look at an environment variable "<PROVIDER>_API_KEY" to use as a default value if unspecified.
func (ProviderOptionAPIKey) Validate ΒΆ
func (p ProviderOptionAPIKey) Validate() error
Validate implements Validatable.
type ProviderOptionModalities ΒΆ
type ProviderOptionModalities Modalities
ProviderOptionModalities is the list of output modalities you request the model to support.
Most provider support text output only. Most models support output of only one modality, either text, image, audio or video. But a few models do support both text and images.
When unspecified, the provider defaults to all modalities supported by the provider and the selected model.
Even when Model is set to a specific model ID, a ListModels call may be made to discover its supported output modalities for providers that support multiple output modalities.
ProviderOptionModalities can be set without a Model to test if a provider supports a modality without causing a ListModels call.
func (ProviderOptionModalities) Validate ΒΆ
func (p ProviderOptionModalities) Validate() error
Validate implements Validatable.
type ProviderOptionModel ΒΆ
type ProviderOptionModel string
ProviderOptionModel specifies which model to use.
For automatic model selection, use the predefined constants ModelCheap, ModelGood, or ModelSOTA directly as provider options. The provider internally calls ListModels() to discover models and select based on its heuristics. Providers that do not support ListModels (e.g. bfl) use a hardcoded list.
When unspecified, no model is selected. This is useful when only calling ListModels(). Generation calls will fail.
Keep in mind that as providers cycle through new models, it's possible a specific model ID is not available anymore or that the default model changes.
const ( // ModelCheap requests the provider to automatically select the cheapest model it can find. ModelCheap ProviderOptionModel = "CHEAP" // ModelGood requests the provider to automatically select a good every day model that has a good // performance/cost trade-off. ModelGood ProviderOptionModel = "GOOD" // ModelSOTA requests the provider to automatically select the best state-of-the-art model // it can find. ModelSOTA ProviderOptionModel = "SOTA" )
Model markers for automatic model selection. These are ProviderOptionModel values that can be passed directly to provider constructors.
func (ProviderOptionModel) Validate ΒΆ
func (p ProviderOptionModel) Validate() error
Validate implements Validatable.
type ProviderOptionPreloadedModels ΒΆ
type ProviderOptionPreloadedModels []Model
ProviderOptionPreloadedModels is a list of models that are preloaded into the provider, to replace the call to ListModels, for example with automatic model selection and modality detection.
This is mostly used for unit tests or repeated client creation to save on HTTP requests.
func (ProviderOptionPreloadedModels) Validate ΒΆ
func (p ProviderOptionPreloadedModels) Validate() error
Validate implements Validatable.
type ProviderOptionRemote ΒΆ
type ProviderOptionRemote string
ProviderOptionRemote is the remote address to access the service.
It is mostly used by locally hosted services (llamacpp, ollama) or for generic client (openaicompatible).
func (ProviderOptionRemote) Validate ΒΆ
func (p ProviderOptionRemote) Validate() error
Validate implements Validatable.
type ProviderOptionStarterWrapper ΒΆ added in v0.4.1
ProviderOptionStarterWrapper wraps the subprocess starter used by CLI-based providers.
This is the subprocess analog of ProviderOptionTransportWrapper: it lets callers intercept process creation for recording, replay, or testing.
func (ProviderOptionStarterWrapper) Validate ΒΆ added in v0.4.1
func (p ProviderOptionStarterWrapper) Validate() error
Validate implements Validatable.
type ProviderOptionTransportWrapper ΒΆ
type ProviderOptionTransportWrapper func(http.RoundTripper) http.RoundTripper
ProviderOptionTransportWrapper wraps the HTTP transport used by the provider.
This is useful for adding middleware like logging, tracing, or HTTP recording for tests.
func (ProviderOptionTransportWrapper) Validate ΒΆ
func (p ProviderOptionTransportWrapper) Validate() error
Validate implements Validatable.
type ProviderPing ΒΆ
type ProviderPing interface {
Provider
// Ping enables confirming that the provider is accessible, without incurring cost. This is useful for local
// providers to detect if they are accessible or not.
Ping(ctx context.Context) error
}
ProviderPing represents a provider that you can ping.
type ProviderScoreboardVariants ΒΆ added in v0.3.0
type ProviderScoreboardVariants interface {
ScoreboardVariants() []ScoreboardVariant
}
ProviderScoreboardVariants is optionally implemented by providers that have multiple regional backends with different model catalogs.
When implemented, documentation tools generate a separate table per variant instead of a single table.
type ProviderUnwrap ΒΆ
type ProviderUnwrap interface {
Unwrap() Provider
}
ProviderUnwrap is exposed when the Provider is actually a wrapper around another one, like adapters.ProviderReasoning or ProviderUsage. This is useful when looking for other interfaces.
type Question ΒΆ added in v0.10.0
type Question struct {
// Type is how the state is evaluated. It selects which of Noul, Choice or Score must be set.
Type QuestionType
// Instructions is what the model should decide or rate. Providers may require it.
Instructions DecisionContent
// Noul describes what the yes and the no outcomes mean. It can only be set for QuestionNoul, where it
// is optional.
//
// Recommended reading: https://docs.typesafe.ai/primitives/noul
Noul *NoulCriteria
// Choice maps the options of a choice question to their descriptions. It can only be set for
// QuestionChoice, where at least one option is required. Use nil for an option that needs no extra
// detail.
//
// Recommended reading: https://docs.typesafe.ai/primitives/choice
Choice map[string]DecisionContent
// Score lists the levels of a score question rubric, in order, starting at level 0. It can only be set
// for QuestionScore, where at least one level is required. Use nil for an undescribed level.
// Providers may impose stricter limits.
//
// Recommended reading: https://docs.typesafe.ai/primitives/score
Score []DecisionContent
}
Question is a typed question about a state.
Exactly one of Noul, Choice or Score can be set, the one matching Type. Instructions is shared by the three types.
Type is explicit instead of inferred from the field that is set: a choice or score question whose criteria are nil at runtime, which append and conditional map building produce, would otherwise be silently asked as a noul question instead of being rejected.
Recommended reading: https://docs.typesafe.ai/primitives/advanced
func (*Question) MarshalJSON ΒΆ added in v0.10.0
MarshalJSON implements json.Marshaler.
type QuestionType ΒΆ added in v0.10.0
type QuestionType string
QuestionType is the type of a Question.
const ( // QuestionNoul is a yes/no question. The answer is the probability that the answer is yes. QuestionNoul QuestionType = "noul" // QuestionChoice picks one option from a set defined by the question. QuestionChoice QuestionType = "choice" // QuestionScore rates the state along an ordered rubric. QuestionScore QuestionType = "score" )
Question types.
type Questions ΒΆ added in v0.10.0
Questions is a set of questions to ask about one state, keyed by the name to report each answer under.
The names are chosen by the caller. They are not sent to the model, they are only used to key the answers.
Set SystemOneRequest.Questions directly. SystemOneResponse.Answers holds the typed answers under the same dynamic names. Entries must not be nil.
Example ΒΆ
ExampleQuestions declares typed questions under caller-selected names.
package main
import (
"encoding/json"
"fmt"
"log"
"github.com/maruel/genai"
)
func main() {
questions := genai.Questions{
"billing": {Type: genai.QuestionNoul, Instructions: genai.Text("Is this request about billing?")},
"tone": {Type: genai.QuestionChoice, Instructions: genai.Text("What is the tone of the customer?"), Choice: map[string]genai.DecisionContent{"calm": nil, "angry": nil}},
}
raw, err := json.MarshalIndent(questions, "", " ")
if err != nil {
log.Print(err)
return
}
fmt.Printf("%s\n", raw)
}
Output: { "billing": { "type": "noul", "instructions": "Is this request about billing?" }, "tone": { "type": "choice", "instructions": "What is the tone of the customer?", "criteria": { "angry": null, "calm": null } } }
type RateLimit ΒΆ
type RateLimit struct {
Type RateLimitType
Period RateLimitPeriod
Limit int64
Remaining int64
Reset time.Time
}
RateLimit contains the limit, remaining, and reset values for a metric.
type RateLimitPeriod ΒΆ
type RateLimitPeriod int32
RateLimitPeriod defines the time period for a rate limit.
const ( PerOther RateLimitPeriod = iota // For non-standard periods PerMinute PerDay PerMonth )
RateLimitPeriod values.
type RateLimitType ΒΆ
type RateLimitType int32
RateLimitType defines the type of rate limit.
const ( Requests RateLimitType = iota + 1 Tokens )
RateLimitType values.
type RealtimeCancel ΒΆ added in v0.11.0
type RealtimeCancel struct {
ResponseID string
}
RealtimeCancel stops generation, optionally targeting ResponseID. It does not remove conversation history or queued playback.
func (RealtimeCancel) Validate ΒΆ added in v0.11.0
func (RealtimeCancel) Validate() error
Validate implements RealtimeInput.
type RealtimeCapabilities ΒΆ added in v0.11.0
type RealtimeCapabilities struct {
AudioCommit bool
CancelResponse bool
ToolResults bool
Truncate bool
Update RealtimeUpdateCapabilities
}
RealtimeCapabilities describes the operations this configured session accepts. A false field means the provider rejects that command with base.ErrNotSupported. Configured features are reported by RealtimeSessionInfo, not here.
AudioCommit reports that RealtimeInputContent.Commit commits manual-turn audio. ToolResults reports that the session accepts tool result batches.
type RealtimeContent ΒΆ added in v0.11.0
type RealtimeContent struct {
// Audio is owned PCM at Info().OutputFormat for Audio events.
Audio []byte
// Text is a delta for Text and transcript events. When Final is true, Text
// is the complete text and replaces accumulated deltas.
Text string
// Reasoning is a delta for Reasoning events.
Reasoning string
// Final marks Text as complete for Text and transcript events.
Final bool
// ToolCalls contains complete calls with validated JSON arguments and unique
// nonempty IDs for ToolCalls events. Providers that omit call IDs must assign
// session-unique IDs and preserve the original provider correlation internally.
ToolCalls []ToolCall
}
RealtimeContent is the payload of a RealtimeEvent. The zero value means the event carries no content. At most one of Audio, Text, or ToolCalls is set. Final is valid only with Text.
func (*RealtimeContent) Validate ΒΆ added in v0.11.0
func (c *RealtimeContent) Validate() error
Validate rejects multiple payload fields or Final without Text.
type RealtimeError ΒΆ added in v0.11.0
type RealtimeError struct {
// Code is the provider error code when reported.
Code string
// Message is a short, caller-safe description of the failure.
Message string
// EventID identifies the rejected client event when reported.
EventID string
}
RealtimeError is an error reported by the provider. RealtimeEventError carries a recoverable one; setup and transport failures are returned directly.
func (*RealtimeError) Error ΒΆ added in v0.11.0
func (e *RealtimeError) Error() string
Error implements error.
type RealtimeEvent ΒΆ added in v0.11.0
type RealtimeEvent struct {
// Kind identifies the event. GenerationDone means output generation ended,
// not playback ended. TurnDone means the provider completed its turn, which
// need not mean session idleness. Interrupted signals the caller to stop
// playback.
Kind RealtimeEventKind
// ResponseID identifies the response within this session. It may be a provider
// identity or an adapter-assigned identity based on provider turn boundaries.
ResponseID string
// ItemID identifies the conversation item when the provider supplies it.
ItemID string
// ContentIndex identifies the content part within ItemID.
ContentIndex int
// Content is the event payload. Its zero value means the event carries no
// content.
Content RealtimeContent
// Usage is a non-cumulative snapshot for one response. Each Usage event
// describes its own response; sum them to compute session usage. ResponseID
// identifies the response when its ownership is known. Limits, when present,
// replaces the previous rate-limit snapshot.
Usage *Usage
// Error carries a recoverable provider error for Error events. The session
// remains usable. Terminal failures are returned by Receive instead.
Error *RealtimeError
// ProviderEvent is provider-specific information for this event. It is nil
// when the provider supplies none. For RealtimeEventProvider, it is the
// complete payload.
ProviderEvent RealtimeProviderEvent
}
RealtimeEvent is one normalized event. Kind identifies the event; Content carries its payload. Provider-specific events use RealtimeEventProvider. Empty IDs mean unspecified. Adapters may assign session-local response identities when documented provider boundaries make correlation unambiguous.
type RealtimeEventKind ΒΆ added in v0.11.0
type RealtimeEventKind string
RealtimeEventKind identifies an event emitted by a session.
const ( // RealtimeEventContentAudio carries one chunk of generated PCM audio at // Info().OutputFormat. Chunks may split words and utterances arbitrarily. RealtimeEventContentAudio RealtimeEventKind = "audio" // RealtimeEventContentText carries generated text. Content.Text is a delta unless // Final is true, in which case it replaces the accumulated deltas. RealtimeEventContentText RealtimeEventKind = "text" // RealtimeEventContentReasoning carries reasoning text that the provider // separates from the answer. Content.Reasoning is a delta. RealtimeEventContentReasoning RealtimeEventKind = "reasoning" // RealtimeEventContentToolCalls carries complete function calls from one response. // Arguments are validated JSON, and IDs are unique and nonempty. The caller // executes the calls and sends the results in RealtimeInputContent.ToolResults. The // turn cannot complete until it receives those results. RealtimeEventContentToolCalls RealtimeEventKind = "tool_calls" // RealtimeEventError reports a recoverable provider error. The session // remains usable. RealtimeEventError RealtimeEventKind = "error" // RealtimeEventGenerationDone reports that the provider ended generation for // one response. Playback may continue, and a response that only calls tools // also emits it. It does not end the turn. An interrupted response also emits // it before TurnDone. RealtimeEventGenerationDone RealtimeEventKind = "generation_done" // RealtimeEventInputSpeechEnd reports that provider voice activity detection // stopped detecting user speech. It occurs only with server turn detection; // manual turns disable provider VAD. ItemID may identify the user item. The // provider may separately commit audio or start a response when configured; // this event guarantees neither. RealtimeEventInputSpeechEnd RealtimeEventKind = "input_speech_end" // RealtimeEventInputSpeechStart reports that provider voice activity detection // detected user speech. It occurs only with server turn detection; manual // turns disable provider VAD. ItemID may identify the user item that speech // will create. It does not guarantee that a turn produces a response. RealtimeEventInputSpeechStart RealtimeEventKind = "input_speech_start" // RealtimeEventInterrupted reports that the provider stopped the active // response before normal completion. Stop and discard queued playback. The // session remains usable. RealtimeEventInterrupted RealtimeEventKind = "interrupted" // RealtimeEventProvider carries an event with no portable kind. ProviderEvent // holds the concrete provider type. Callers must ignore provider event types // they do not handle. RealtimeEventProvider RealtimeEventKind = "provider" // RealtimeEventTranscriptInput carries a transcription of user audio. // Content.Text is a delta unless Final is true, in which case it replaces the // accumulated deltas. Transcription is independent of model output and may // arrive out of order with respect to response events. RealtimeEventTranscriptInput RealtimeEventKind = "input_transcript" // RealtimeEventTranscriptOutput carries a transcription of generated audio. // Content.Text is a delta unless Final is true, in which case it replaces the // accumulated deltas. It is not generated text output; RealtimeEventContentText // carries that. RealtimeEventTranscriptOutput RealtimeEventKind = "output_transcript" // RealtimeEventTurnDone reports that the provider completed the current turn. // No more output follows for the current input unless the caller submits tool // results. It also follows an interrupted, cancelled, or failed response. The // session remains open. RealtimeEventTurnDone RealtimeEventKind = "turn_done" // RealtimeEventUsage reports token usage for one response and may include a // rate-limit snapshot. It is not cumulative; sum the token fields across // events for session totals. When Usage.Limits is set, it replaces the // previous limits. RealtimeEventUsage RealtimeEventKind = "usage" )
RealtimeEventKind values. Each kind defines the RealtimeEvent fields that apply to it.
type RealtimeInput ΒΆ added in v0.11.0
type RealtimeInput interface {
Validate() error
// contains filtered or unexported methods
}
RealtimeInput is one command sent to RealtimeSession. Only the types in this package implement it.
type RealtimeInputContent ΒΆ added in v0.11.0
type RealtimeInputContent struct {
// Audio appends PCM at Info().InputFormat. It may be empty only with Commit.
Audio []byte
// Doc adds a user turn carrying the document. Text may follow it.
Doc Doc
// Text adds a user text turn. It may be empty only with Generate.
Text string
// ToolResults answers completed calls. It always requests a response.
ToolResults []ToolCallResult
// Commit commits buffered audio. Alone it does not request a reply; with
// Generate it commits before requesting one.
Commit bool
// Generate requests a response. It commits nonempty audio first.
Generate bool
}
RealtimeInputContent sends one content command. Audio is selected by Audio or Commit; otherwise one of Doc or ToolResults is set, and Text may accompany Doc as one turn. The zero value is invalid.
Audio appends PCM at Info().InputFormat to the buffered user audio. Commit commits the buffer and may close the provider audio stream; an empty Commit commits audio sent earlier. Generate requests a reply and commits nonempty audio first when needed. Commit and Generate may be combined: Commit finalizes buffered audio, then Generate requests the response. With server turn detection, the provider may ignore Generate and reply on its own.
Doc adds a user turn carrying the document. Text, when set, follows it in the same turn and Generate requests the response. Each provider bounds the document size. A model that turns the document down reports a RealtimeEventError for the command.
Text adds a user text turn, or follows the document when Doc is set. Generate=false adds passive context; Generate=true submits the turn and requests a response. An empty Text with Generate=true requests a response from the current context. Providers without an explicit generation command may ignore it.
ToolResults answers completed calls. Send complete batches to preserve parallel call order. Tool results always request a response; Generate is ignored for this payload.
func (RealtimeInputContent) Validate ΒΆ added in v0.11.0
func (i RealtimeInputContent) Validate() error
Validate checks that one payload is set and that its flags are valid.
type RealtimeOption ΒΆ added in v0.11.0
type RealtimeOption interface {
Validate() error
}
RealtimeOption configures a realtime session.
Portable options are RealtimeOptionInstructions, RealtimeOptionAudio, RealtimeOptionTools, RealtimeTurnDetection, and Modalities. Modalities selects output modalities and accepts only ModalityAudio and ModalityText; empty uses the provider's default output mode. Input is per command: audio, a document, text, or tool results.
A provider may define its own option type for settings that have no portable meaning. A provider rejects options it cannot honor rather than ignoring them.
type RealtimeOptionAudio ΒΆ added in v0.11.0
RealtimeOptionAudio requests voice selection and audio transcripts. Transcription is opt-in and independent of generated text output. A false transcription field requests no transcription but does not promise that the provider sends no transcript events; callers must handle them either way.
func (*RealtimeOptionAudio) Validate ΒΆ added in v0.11.0
func (*RealtimeOptionAudio) Validate() error
Validate implements RealtimeOption.
type RealtimeOptionInstructions ΒΆ added in v0.11.0
type RealtimeOptionInstructions string
RealtimeOptionInstructions supplies session instructions.
func (RealtimeOptionInstructions) Validate ΒΆ added in v0.11.0
func (RealtimeOptionInstructions) Validate() error
Validate implements RealtimeOption.
type RealtimeOptionTools ΒΆ added in v0.11.0
type RealtimeOptionTools struct {
Tools []ToolDef
}
RealtimeOptionTools declares session tools. The application owns their execution.
func (*RealtimeOptionTools) Validate ΒΆ added in v0.11.0
func (o *RealtimeOptionTools) Validate() error
Validate implements RealtimeOption.
type RealtimeProviderEvent ΒΆ added in v0.11.0
type RealtimeProviderEvent interface {
// RealtimeEventName returns the provider's wire event name.
RealtimeEventName() string
}
RealtimeProviderEvent is a provider-specific event payload. Providers define the concrete types and document the event names they represent. Providers use it when no portable event kind applies; callers ignore event kinds they do not handle.
type RealtimeSession ΒΆ added in v0.11.0
type RealtimeSession interface {
io.Closer
// Info returns immutable native formats, accepted input kinds, configured
// features, and accepted operations after setup.
Info() RealtimeSessionInfo
// Send applies one complete command to the session and writes it to the
// provider. It rejects invalid, unsupported, and out-of-order commands before
// the first wire write. It serializes concurrent calls, translates the command
// into the provider's wire messages, and updates local correlation only after
// every frame is written. Input buffers are borrowed until Send returns and are
// not retained. A successful return means written, not accepted by the
// provider. A write failure may leave a partial transaction and terminates the
// session; do not retry it.
Send(ctx context.Context, input RealtimeInput) error
// Receive returns the events as they arrive from the server. Slow consumers
// apply backpressure; providers must bound frame sizes and any pending events
// rather than queue indefinitely. Unknown wire events may be skipped or returned
// as RealtimeEventProvider events; malformed recognized events are errors. Yielded
// content must satisfy RealtimeContent.Validate. The sequence stops after the
// first error, including io.EOF for normal peer closure. Breaking out stops
// receiving without closing the session. Canceling ctx terminates the transport.
Receive(ctx context.Context) iter.Seq2[RealtimeEvent, error]
}
RealtimeSession is one provider-owned conversation and its transport.
One goroutine receives events while other goroutines may call Send. Concurrent sends are serialized; callers must order dependent commands themselves. Close may run concurrently and wakes blocked Send and Receive calls. Call it once. The session does not reconnect, replay, execute tools, detect voice activity, or resample.
Example (ManualTurns) ΒΆ
package main
import (
"context"
"fmt"
"log"
"os"
"github.com/maruel/genai"
"github.com/maruel/genai/providers/llamacpp"
)
func main() {
ctx := context.Background()
// Run llama-server with a native audio model. The loaded model owns the
// input and output sample rates. See the llamacppsrv package for a managed
// runtime.
c, err := llamacpp.New(ctx, genai.ProviderOptionRemote("http://127.0.0.1:8080"))
if err != nil {
log.Print(err)
return
}
defer func() {
if err := c.Close(); err != nil {
log.Printf("Close: %v", err)
}
}()
s, err := c.ConnectRealtime(ctx, &llamacpp.RealtimeOption{MaxOutputTokens: 256})
if err != nil {
log.Print(err)
return
}
defer func() {
if err := s.Close(); err != nil {
log.Printf("Close: %v", err)
}
}()
// A live application captures the microphone at s.Info().InputFormat.
// This recorded turn is mono PCM16LE at the model's native input rate.
pcm, err := os.ReadFile("question-16k.pcm")
if err != nil {
log.Print(err)
return
}
if err := s.Send(ctx, genai.RealtimeInputContent{Audio: pcm, Commit: true, Generate: true}); err != nil {
log.Print(err)
return
}
// GenerationDone is not terminal: the provider may emit filler output or a
// tool call before TurnDone. Keep receiving until TurnDone.
for e, err := range s.Receive(ctx) {
if err != nil {
log.Print(err)
return
}
switch e.Kind {
case genai.RealtimeEventContentAudio:
// Play e.Content.Audio at s.Info().OutputFormat.
case genai.RealtimeEventContentText:
fmt.Print(e.Content.Text)
case genai.RealtimeEventContentToolCalls:
results := make([]genai.ToolCallResult, len(e.Content.ToolCalls))
for i := range e.Content.ToolCalls {
results[i] = genai.ToolCallResult{ID: e.Content.ToolCalls[i].ID, Name: e.Content.ToolCalls[i].Name, Result: `{"ok":true}`}
}
if err := s.Send(ctx, genai.RealtimeInputContent{ToolResults: results, Generate: true}); err != nil {
log.Print(err)
return
}
case genai.RealtimeEventTurnDone:
// TurnDone is terminal: no content or tool call follows it.
return
default:
}
}
}
Output:
Example (ServerTurns) ΒΆ
package main
import (
"context"
"fmt"
"log"
"github.com/maruel/genai"
"github.com/maruel/genai/providers/openaichat"
)
func main() {
ctx := context.Background()
// llama.cpp supports manual turns only. Use a provider with server turn
// detection, such as OpenAI Realtime, for this mode.
c, err := openaichat.New(ctx, genai.ProviderOptionModel("gpt-realtime-2.1"))
if err != nil {
log.Print(err)
return
}
defer func() {
if err := c.Close(); err != nil {
log.Printf("Close: %v", err)
}
}()
s, err := c.ConnectRealtime(ctx, &genai.RealtimeOptionAudio{OutputTranscription: true})
if err != nil {
log.Print(err)
return
}
defer func() {
if err := s.Close(); err != nil {
log.Printf("Close: %v", err)
}
}()
// Send a text turn. A microphone loop would send Audio at
// s.Info().InputFormat instead and let the provider detect the turn end.
if err := s.Send(ctx, genai.RealtimeInputContent{Text: "Say hello in one word.", Generate: true}); err != nil {
log.Print(err)
return
}
// GenerationDone is not terminal. Continue until TurnDone, which marks the
// end of the turn and is never followed by content or tool calls.
for e, err := range s.Receive(ctx) {
if err != nil {
log.Print(err)
return
}
switch e.Kind {
case genai.RealtimeEventContentAudio:
// Play e.Content.Audio at s.Info().OutputFormat.
case genai.RealtimeEventTranscriptOutput:
fmt.Print(e.Content.Text)
case genai.RealtimeEventError:
log.Printf("Provider rejected a command: %v", e.Error)
case genai.RealtimeEventInterrupted:
// Stop playback and discard queued audio.
case genai.RealtimeEventTurnDone:
// TurnDone is terminal: no content or tool call follows it.
return
default:
}
}
}
Output:
type RealtimeSessionInfo ΒΆ added in v0.11.0
type RealtimeSessionInfo struct {
ID string
InputFormat AudioFormat
OutputFormat AudioFormat
// InputModalities lists the payload kinds the session encodes: ModalityAudio,
// ModalityImage, and ModalityText. It states what the transport accepts.
InputModalities []Modality
// TurnDetection is the accepted server or manual turn mode.
TurnDetection RealtimeTurnDetection
Capabilities RealtimeCapabilities
}
RealtimeSessionInfo contains the accepted session identity, native formats, accepted input kinds, turn mode, and capabilities. IDs are provider-supplied when available; an empty ID means none was supplied.
type RealtimeToolCallCancelledError ΒΆ added in v0.11.0
type RealtimeToolCallCancelledError struct {
// ToolCallIDs are the cancelled calls matched in the rejected batch.
ToolCallIDs []string
}
RealtimeToolCallCancelledError means a tool result batch matched one or more calls the provider cancelled. The batch was rejected locally without wire effects. Callers may discard the listed calls and retry the remaining results.
func (*RealtimeToolCallCancelledError) Error ΒΆ added in v0.11.0
func (e *RealtimeToolCallCancelledError) Error() string
Error implements error.
type RealtimeTruncate ΒΆ added in v0.11.0
RealtimeTruncate reconciles unheard output audio in ItemID at ContentIndex and AudioEndMS. It requires Capabilities.Truncate and does not stop generation.
func (RealtimeTruncate) Validate ΒΆ added in v0.11.0
func (i RealtimeTruncate) Validate() error
Validate checks the truncation target.
type RealtimeTurnDetection ΒΆ added in v0.11.0
type RealtimeTurnDetection string
RealtimeTurnDetection selects who decides that microphone input is complete.
const ( RealtimeTurnManual RealtimeTurnDetection = "manual" RealtimeTurnServer RealtimeTurnDetection = "server" )
RealtimeTurnDetection values. Empty selects the provider's documented default.
func (RealtimeTurnDetection) Validate ΒΆ added in v0.11.0
func (o RealtimeTurnDetection) Validate() error
Validate implements RealtimeOption.
type RealtimeUpdate ΒΆ added in v0.11.0
type RealtimeUpdate struct {
Instructions *string
Tools *[]ToolDef
ToolChoice *ToolCallRequest
MaxOutputTokens *int64
OutputModalities *Modalities
TurnDetection *RealtimeTurnDetection
Voice *string
}
RealtimeUpdate changes live session settings. A nil field is unchanged. Tools may be an empty list to clear declarations. Instructions may be empty to clear them. MaxOutputTokens must be positive. TurnDetection selects manual or server turns. Voice changes only before output audio on providers that report VoiceBeforeOutput.
func (RealtimeUpdate) Validate ΒΆ added in v0.11.0
func (u RealtimeUpdate) Validate() error
Validate rejects empty, malformed, or ambiguous updates.
type RealtimeUpdateCapabilities ΒΆ added in v0.11.0
type RealtimeUpdateCapabilities struct {
Instructions bool
Tools bool
ToolChoice bool
ToolChoiceRequired bool
MaxOutputTokens bool
OutputModalities bool
TurnDetection bool
VoiceBeforeOutput bool
}
RealtimeUpdateCapabilities describes the session settings this configured session accepts at runtime. ToolChoice covers auto and none; required has its own field because some providers can clear tools but cannot force a call. VoiceBeforeOutput rejects a voice change after the session produced audio.
type ReasoningOutput ΒΆ added in v0.11.0
type ReasoningOutput struct{}
ReasoningOutput requests optional reasoning or summaries for generation and realtime sessions. Omitting it selects the provider's cheapest output mode. Returned reasoning and native continuation state are always preserved.
var IncludeReasoning ReasoningOutput
IncludeReasoning requests optional reasoning where available. It does not enable thinking or guarantee that the model returns reasoning.
func (ReasoningOutput) Validate ΒΆ added in v0.11.0
func (ReasoningOutput) Validate() error
Validate implements GenOption and RealtimeOption.
type Reply ΒΆ
type Reply struct {
// Text is the content of the text message.
Text string `json:"text,omitzero"`
// Doc can be audio, video, image, PDF or any other format, including reference text.
Doc Doc `json:"doc,omitzero"`
// Citation contains references to source material that support the content.
Citation Citation `json:"citation,omitzero"`
// Reasoning is the reasoning done by the LLM. Keep it in conversation history;
// some models require it when continuing a turn after tool calls.
Reasoning string `json:"reasoning,omitzero"`
// ToolCall is a tool call that the LLM requested to make.
ToolCall ToolCall `json:"tool_call,omitzero"`
// Opaque is added to keep continuity on the processing. A good example is Anthropic's extended thinking, or
// server-side tool calling. It must be kept during an exchange.
//
// A message with only Opaque set is valid. It can be used in combination with other fields. This field is
// specific to both the provider and the model.
//
// The data must be JSON-serializable.
Opaque map[string]any `json:"opaque,omitzero"`
// contains filtered or unexported fields
}
Reply is a block of information returned by the provider.
Normally only one of the field must be set. The exception is the Opaque field.
Reply generally represents content returned by the provider, like a block of text or a document returned by the model. It can be a silent tool call request. It can also be an opaque block. A good example is traces of server side tool calling like Web search or MCP tool calling.
func (*Reply) GoString ΒΆ
GoString returns a JSON representation of the reply for debugging purposes.
func (*Reply) UnmarshalJSON ΒΆ
UnmarshalJSON implements the json.Unmarshaler interface.
type Request ΒΆ
type Request struct {
// Text is the content of the text message.
Text string `json:"text,omitzero"`
// Doc can be audio, video, image, PDF or any other format, including reference text.
Doc Doc `json:"doc,omitzero"`
// contains filtered or unexported fields
}
Request is an input content block for generation or embedding.
It is effectively a union, only one of the 2 related field groups can be set.
func (*Request) UnmarshalJSON ΒΆ
UnmarshalJSON implements the json.Unmarshaler interface.
type Result ΒΆ
type Result struct {
Message
Usage Usage
// Logprobs is a list of multiple log probabilities, each for a token.
//
// The first item of each subslice is the chosen token. The next items are the candidates not chosen.
//
// Some providers only return the probability for the chosen tokens and not for the candidates.
Logprobs [][]Logprob
}
Result is the result of a completion.
It is a Message along with Usage metadata about the operation. It optionally include Logprobs if requested and the provider support it.
type ScoreLegend ΒΆ added in v0.10.0
type ScoreLegend map[string]DecisionContent
ScoreLegend is the description of each level of a score answer, keyed by the level.
It is the Question.Score criteria of the question, echoed back by the API. The values are decoded as Text, Object or Array; a nil value is a level the caller left undescribed.
func (*ScoreLegend) UnmarshalJSON ΒΆ added in v0.10.0
func (l *ScoreLegend) UnmarshalJSON(b []byte) error
UnmarshalJSON implements json.Unmarshaler.
DecisionContent is an interface, so encoding/json cannot decode it on its own; each level is decoded into the variant matching its JSON token here.
type ScoreboardVariant ΒΆ added in v0.3.0
type ScoreboardVariant struct {
// Name is the display name for this variant, e.g. "Intl", "US".
Name string
// Score is the scoreboard data for this variant.
Score scoreboard.Score
}
ScoreboardVariant is a named scoreboard for a specific backend or region of a provider.
type Starter ΒΆ added in v0.4.1
type Starter func(ctx context.Context, args []string) (stdin io.WriteCloser, stdout io.ReadCloser, wait func() error, err error)
Starter launches a subprocess and returns its stdin, stdout, a wait function, and any error.
It is the subprocess equivalent of http.RoundTripper: the lowest-level abstraction that CLI-based providers (claudecode, codex, opencode) use to spawn a process.
type SystemOneRequest ΒΆ added in v0.10.0
type SystemOneRequest struct {
// State is required unless Docs contains one application/json document.
State DecisionContent `json:"state"`
Questions Questions `json:"questions"`
// Docs contains inline document attachments. When State is unset, one application/json document
// supplies the state instead. Supported attachment types depend on the provider and model.
// Remote URLs are not supported.
Docs []Doc `json:"docs,omitzero"`
}
SystemOneRequest is a System One decision request. Providers use their client's configured model and own transport envelopes and stricter protocol limits. Docs are inline attachments. Providers select supported document types, enforce read limits, and encode document bytes in their native wire format.
func (*SystemOneRequest) ReadState ΒΆ added in v0.10.0
func (r *SystemOneRequest) ReadState(maxDocBytes int64) (DecisionContent, []Doc, error)
ReadState validates the request and returns the resolved state and remaining attachments. It reads a JSON state document with maxDocBytes as the byte limit, but does not read attachments. It leaves request fields unchanged. Returned attachments share the request's Docs slice.
func (*SystemOneRequest) Validate ΒΆ added in v0.10.0
func (r *SystemOneRequest) Validate() error
Validate checks the shared state, questions and inline documents without reading attachment bytes. Providers validate document types.
type SystemOneResponse ΒΆ added in v0.10.0
type SystemOneResponse struct {
// Answers holds one answer per question, keyed by the question name.
Answers Answers `json:"answers"`
// Usage is the token usage of the request.
Usage DecisionUsage `json:"usage"`
}
SystemOneResponse is the response of a POST /v1/systemone request.
func (*SystemOneResponse) ValidateQuestions ΒΆ added in v0.10.0
func (r *SystemOneResponse) ValidateQuestions(qs *Questions) error
ValidateQuestions checks that the response answers each requested question with the right type.
type ToolCall ΒΆ
type ToolCall struct {
ID string `json:"id,omitzero"` // Unique identifier for the tool call. Necessary for parallel tool calling.
Name string `json:"name,omitzero"` // Tool being called.
Arguments string `json:"arguments,omitzero"` // encoded as JSON
// Opaque is added to keep continuity on the processing. A good example is Anthropic's extended thinking. It
// must be kept during an exchange.
//
// A message with only Opaque set is valid.
Opaque map[string]any `json:"opaque,omitzero"`
// contains filtered or unexported fields
}
ToolCall is a tool call that the LLM requested to make.
func (*ToolCall) Call ΒΆ
Call invokes the ToolDef.Callback with arguments from the ToolCall, returning the result string.
It decodes the ToolCall.Arguments and passes it to the ToolDef.Callback.
func (*ToolCall) UnmarshalJSON ΒΆ
UnmarshalJSON implements the json.Unmarshaler interface.
type ToolCallRequest ΒΆ
type ToolCallRequest int
ToolCallRequest determines if we want the LLM to request a tool call.
const ( // ToolCallAny is the default, the model is free to choose if a tool is called or not. For some models (like // llama family), it may be a bit too "tool call happy". ToolCallAny ToolCallRequest = iota // ToolCallRequired means a tool call is required. Don't forget to change the value after sending the // response! ToolCallRequired // ToolCallNone means that while tools are described, they should not be called. It is useful when a LLM did // tool calls, got the response and now it's time to generate some text to present to the end user. ToolCallNone )
type ToolCallResult ΒΆ
type ToolCallResult struct {
ID string `json:"id,omitzero"`
Name string `json:"name,omitzero"`
Result string `json:"result,omitzero"`
// contains filtered or unexported fields
}
ToolCallResult is the result for a tool call that the LLM requested to make.
func (*ToolCallResult) UnmarshalJSON ΒΆ
func (t *ToolCallResult) UnmarshalJSON(b []byte) error
UnmarshalJSON implements the json.Unmarshaler interface.
func (*ToolCallResult) Validate ΒΆ
func (t *ToolCallResult) Validate() error
Validate ensures the tool result is valid.
type ToolDef ΒΆ
type ToolDef struct {
// Name must be unique among all tools.
Name string
// Description must be a LLM-friendly short description of the tool.
Description string
// Callback is the function to call with the inputs.
// It must accept a context.Context one struct pointer as input: (ctx context.Context, input *struct{}). The
// struct must use json_schema to be serializable as JSON.
// It must return the result and an error: (string, error).
Callback any
// InputSchemaOverride overrides the schema deduced from the Callback's second argument. It's meant to be
// used when an enum or a description is set dynamically, or with complex if/then/else that would be tedious
// to describe as struct tags.
//
// It is okay to initialize Callback, then take the return value of GetInputSchema() to initialize InputSchemaOverride, then mutate it.
InputSchemaOverride JSONSchema
// contains filtered or unexported fields
}
ToolDef describes a tool that the LLM can request to use.
func (*ToolDef) GetInputSchema ΒΆ
func (t *ToolDef) GetInputSchema() (JSONSchema, error)
GetInputSchema returns the json schema for the input argument of the callback.
func (*ToolDef) Validate ΒΆ
Validate ensures the tool definition is valid.
For the Name field, it uses the rule according to https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/implement-tool-use#example-simple-tool-definition
type Usage ΒΆ
type Usage struct {
// Token usage for the current request.
InputTokens int64
InputCachedTokens int64
ReasoningTokens int64
OutputTokens int64
TotalTokens int64
// ContextTokens is the number of tokens in the context window after the
// request. Zero when the provider does not report it.
ContextTokens int64
// ContextWindow is the maximum number of tokens the context window holds.
// Zero when the provider does not report it.
ContextWindow int64
// FinishReason indicates why the model stopped generating tokens.
FinishReason FinishReason
// ServiceTier is the quality of service tier used to process the request,
// as reported by the provider. Common values include "standard", "batch",
// "flex", "default", "auto", etc. Empty when not reported.
ServiceTier string
// Limits contains a list of rate limit details from the provider.
Limits []RateLimit
}
Usage from the LLM provider.
Directories
ΒΆ
| Path | Synopsis |
|---|---|
|
Package acp implements a client for the Agent Client Protocol (ACP).
|
Package acp implements a client for the Agent Client Protocol (ACP). |
|
Package adapters provides adapters for genai providers and realtime sessions.
|
Package adapters provides adapters for genai providers and realtime sessions. |
|
Package base provides shared infrastructure for implementing genai providers.
|
Package base provides shared infrastructure for implementing genai providers. |
|
cmd
|
|
|
cache-mgr
command
Command cache-mgr fetches and prints out the list of files stored on the selected provider.
|
Command cache-mgr fetches and prints out the list of files stored on the selected provider. |
|
list-models
command
Command list-models fetches and prints out the list of models from the selected providers.
|
Command list-models fetches and prints out the list of models from the selected providers. |
|
llama-serve
command
Command llama-serve fetches a model from HuggingFace and runs llama-server.
|
Command llama-serve fetches a model from HuggingFace and runs llama-server. |
|
scoreboard
command
Command scoreboard generates a scoreboard for every providers supported.
|
Command scoreboard generates a scoreboard for every providers supported. |
|
examples
|
|
|
aud-img-txt_to_embed
command
|
|
|
aud-txt_to_txt
command
|
|
|
img-txt_to_decisions_local
command
|
|
|
img-txt_to_img
command
|
|
|
img-txt_to_img-txt
command
|
|
|
img-txt_to_txt
command
|
|
|
img-txt_to_txt_local
command
|
|
|
img-txt_to_vid
command
|
|
|
txt_to_decisions
command
|
|
|
txt_to_decisions_local
command
|
|
|
txt_to_img
command
|
|
|
txt_to_txt_any
command
|
|
|
txt_to_txt_citations
command
|
|
|
txt_to_txt_decode-json
command
|
|
|
txt_to_txt_logprobs
command
|
|
|
txt_to_txt_oauth2_codex
command
Authenticate with Codex's saved ChatGPT subscription and call GPT-6 Luna directly over HTTP, without launching Codex.
|
Authenticate with Codex's saved ChatGPT subscription and call GPT-6 Luna directly over HTTP, without launching Codex. |
|
txt_to_txt_oauth2_gemini
command
|
|
|
txt_to_txt_quota
command
|
|
|
txt_to_txt_stream
command
|
|
|
txt_to_txt_sync
command
|
|
|
txt_to_txt_sync_multi
command
|
|
|
txt_to_txt_thinking
command
|
|
|
txt_to_txt_tool-manual
command
|
|
|
txt_to_txt_tool-stream
command
|
|
|
txt_to_txt_tool-sync
command
|
|
|
txt_to_txt_websearch-stream
command
|
|
|
txt_to_txt_websearch-sync
command
|
|
|
vid-txt_to_txt
command
|
|
|
Package httprecord provides safe HTTP recording logic for users that was to understand the API and do smoke tests.
|
Package httprecord provides safe HTTP recording logic for users that was to understand the API and do smoke tests. |
|
Package internal provides shared utilities for the genai library.
|
Package internal provides shared utilities for the genai library. |
|
bb
Package bb is a separate package so it can be imported by genai while being internal and exported so cmp.Diff() isn't unhappy.
|
Package bb is a separate package so it can be imported by genai while being internal and exported so cmp.Diff() isn't unhappy. |
|
cli
Package cli provides process helpers for CLI-based providers.
|
Package cli provides process helpers for CLI-based providers. |
|
cmd/autofix-weekly-regen
command
Command autofix-weekly-regen asks pi.dev to repair weekly model regeneration failures.
|
Command autofix-weekly-regen asks pi.dev to repair weekly model regeneration failures. |
|
cmd/gen-realtimespeech
command
Command gen-realtimespeech regenerates the realtime speech FLAC fixtures with llama-tts.
|
Command gen-realtimespeech regenerates the realtime speech FLAC fixtures with llama-tts. |
|
cmd/update-servers
command
Command update-servers checks for newer releases of llama.cpp and ollama servers used by llamacppsrv and ollamasrv, updates the version constants, and regenerates test recordings.
|
Command update-servers checks for newer releases of llama.cpp and ollama servers used by llamacppsrv and ollamasrv, updates the version constants, and regenerates test recordings. |
|
ghrelease
Package ghrelease provides shared helpers for downloading and extracting GitHub release assets.
|
Package ghrelease provides shared helpers for downloading and extracting GitHub release assets. |
|
internaltest
Package internaltest is awesome sauce for unit testing.
|
Package internaltest is awesome sauce for unit testing. |
|
msgutil
Package msgutil provides shared message utilities for subprocess-based providers.
|
Package msgutil provides shared message utilities for subprocess-based providers. |
|
myrecorder
Package myrecorder has HTTP recording logic.
|
Package myrecorder has HTTP recording logic. |
|
realtime
Package realtime provides cancellable WebSocket I/O with bounded reads for realtime providers.
|
Package realtime provides cancellable WebSocket I/O with bounded reads for realtime providers. |
|
realtimetest
Package realtimetest provides scripted WebSocket peers, session checks and a fake session.
|
Package realtimetest provides scripted WebSocket peers, session checks and a fake session. |
|
sse
Package sse provides Server-Sent Events (SSE) processing utilities.
|
Package sse provides Server-Sent Events (SSE) processing utilities. |
|
Package providers is the root of all standard providers.
|
Package providers is the root of all standard providers. |
|
alibaba
Package alibaba implements a client for the Alibaba Cloud DashScope API (Model Studio).
|
Package alibaba implements a client for the Alibaba Cloud DashScope API (Model Studio). |
|
anthropic
Package anthropic implements a client for the Anthropic API, to use Claude.
|
Package anthropic implements a client for the Anthropic API, to use Claude. |
|
antigravity
Package antigravity implements a genai provider backed by the Google Antigravity CLI (`agy`).
|
Package antigravity implements a genai provider backed by the Google Antigravity CLI (`agy`). |
|
antigravity/internal/extracttypes
command
Command extracttypes prints the struct types, with field tags, embedded in a stripped Go binary.
|
Command extracttypes prints the struct types, with field tags, embedded in a stripped Go binary. |
|
baseten
Package baseten implements a client for the Baseten inference API.
|
Package baseten implements a client for the Baseten inference API. |
|
bfl
Package bfl implements a client for Black Forest Labs API.
|
Package bfl implements a client for Black Forest Labs API. |
|
cerebras
Package cerebras implements a client for the Cerebras API.
|
Package cerebras implements a client for the Cerebras API. |
|
claudecode
Package claudecode implements a genai provider backed by the Claude Code CLI.
|
Package claudecode implements a genai provider backed by the Claude Code CLI. |
|
cloudflare
Package cloudflare implements a client for the Cloudflare AI API.
|
Package cloudflare implements a client for the Cloudflare AI API. |
|
codex
Package codex implements a genai provider backed by the Codex CLI.
|
Package codex implements a genai provider backed by the Codex CLI. |
|
cohere
Package cohere implements a client for the Cohere API.
|
Package cohere implements a client for the Cohere API. |
|
deepseek
Package deepseek implements a client for the DeepSeek API.
|
Package deepseek implements a client for the DeepSeek API. |
|
dsh
Package dsh implements a genai provider backed by the DeepSeek Harness CLI.
|
Package dsh implements a genai provider backed by the DeepSeek Harness CLI. |
|
gemini
Package gemini implements a client for Google's Gemini API.
|
Package gemini implements a client for Google's Gemini API. |
|
groq
Package groq implements a client for the Groq API.
|
Package groq implements a client for the Groq API. |
|
huggingface
Package huggingface implements a client for the HuggingFace serverless inference API.
|
Package huggingface implements a client for the HuggingFace serverless inference API. |
|
llamacpp
Package llamacpp implements a client for llama-server native APIs.
|
Package llamacpp implements a client for llama-server native APIs. |
|
llamacpp/llamacppsrv
Package llamacppsrv downloads llama.cpp executables and starts llama-server.
|
Package llamacppsrv downloads llama.cpp executables and starts llama-server. |
|
mistral
Package mistral implements a client for the Mistral API.
|
Package mistral implements a client for the Mistral API. |
|
ollama
Package ollama implements a client for the Ollama API.
|
Package ollama implements a client for the Ollama API. |
|
ollama/ollamasrv
Package ollamasrv downloads and starts ollama directly from GitHub releases.
|
Package ollamasrv downloads and starts ollama directly from GitHub releases. |
|
openaibase
Package openaibase contains shared types and client operations used by the OpenAI Chat Completion, Responses and Realtime providers, and by compatible providers such as llama.cpp.
|
Package openaibase contains shared types and client operations used by the OpenAI Chat Completion, Responses and Realtime providers, and by compatible providers such as llama.cpp. |
|
openaichat
Package openaichat implements a client for the OpenAI Chat Completion API.
|
Package openaichat implements a client for the OpenAI Chat Completion API. |
|
openaicompatible
Package openaicompatible implements a lenient client for "OpenAI-compatible" providers.
|
Package openaicompatible implements a lenient client for "OpenAI-compatible" providers. |
|
openairesponses
Package openairesponses implements a client for the OpenAI Responses API.
|
Package openairesponses implements a client for the OpenAI Responses API. |
|
opencode
Package opencode implements a genai provider backed by the OpenCode CLI.
|
Package opencode implements a genai provider backed by the OpenCode CLI. |
|
openrouter
Package openrouter implements a client for the OpenRouter API.
|
Package openrouter implements a client for the OpenRouter API. |
|
pi
Package pi implements a genai provider backed by the Pi coding agent CLI.
|
Package pi implements a genai provider backed by the Pi coding agent CLI. |
|
pollinations
Package pollinations implements a client for the Pollinations API.
|
Package pollinations implements a client for the Pollinations API. |
|
togetherai
Package togetherai implements a client for the Together.ai API.
|
Package togetherai implements a client for the Together.ai API. |
|
typesafe
Package typesafe implements a client for the TypeSafe System One API, to use Jev.
|
Package typesafe implements a client for the TypeSafe System One API, to use Jev. |
|
xiaomi
Package xiaomi implements a client for the Xiaomi MiMo platform API.
|
Package xiaomi implements a client for the Xiaomi MiMo platform API. |
|
Package scoreboard declares the structures to define a scoreboard.
|
Package scoreboard declares the structures to define a scoreboard. |
|
Package smoke defines probe families and the recorder they measure through.
|
Package smoke defines probe families and the recorder they measure through. |
|
smoketest
Package smoketest runs a scoreboard in test mode.
|
Package smoketest runs a scoreboard in test mode. |
|
Package subprocessrecord provides recording and replay of subprocess I/O for CLI-based providers (claudecode, codex, dsh, opencode).
|
Package subprocessrecord provides recording and replay of subprocess I/O for CLI-based providers (claudecode, codex, dsh, opencode). |
|
Package websocketrecord provides recording and replay of WebSocket message exchanges for provider smoke tests.
|
Package websocketrecord provides recording and replay of WebSocket message exchanges for provider smoke tests. |




