genai

package module
v0.11.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 9, 2026 License: Apache-2.0 Imports: 21 Imported by: 18

README ΒΆ

genai

The opinionated high performance professional-grade AI package for Go.

genai is intentional. Curious why it was created? See the release announcement at maruel.ca/post/genai-v0.1.0.

Go Reference codecov

Features

  • Full functionality: Full access to each backend-specific functionality. Access the raw API if needed with full message schema as Go structs.
  • Tool calling via reflection: Tell the LLM to call a tool directly, described as a Go struct. No need to manually fiddle with JSON.
  • Native JSON struct serialization: Pass a struct to tell the LLM what to generate, decode the reply into your struct. No need to manually fiddle with JSON. Supports required fields, enums, descriptions, etc. You can still fiddle if you want to. :)
  • Decision inference: Ask typed questions about text or JSON state through Provider.SystemOne(). TypeSafe's Jev, Cloudflare's CLEF and CLEF-Flash, and local llama.cpp decision models share genai.Questions and genai.Answers maps with dynamic names and typed values. Document support depends on the provider and model. See the TypeSafe, Cloudflare, and llama.cpp examples.
  • Streaming: Streams completion reply as the output is being generated, including thinking and tool calling, via go 1.23 iterators.
  • Multi-modal: Process images, PDFs and videos (!) as input or output.
  • Web Search: Search the web to answer your question and cite documents passed in.
  • Smoke testing friendly: record and play back API calls at HTTP level to save πŸ’° and keep tests fast and reproducible, via the exposed HTTP transport. See example.
  • Rate limits and usage: Parse the provider-specific HTTP headers and JSON response to get the tokens usage and remaining quota.
  • Provide access to HTTP headers to enable beta features.

Design

  • Safe and strict API implementation. All you love from a statically typed language. The library's smoke tests immediately fail on unknown RPC fields. Error code paths are properly implemented.
  • Stateless. No global state, it is safe to use clients concurrently.
  • Professional grade. smoke tested on live services with recorded traces located in testdata/ directories, e.g. providers/anthropic/testdata/TestClient/Scoreboard/.
  • Trust, but verify. It generates a scoreboard based on actual behavior from each provider.
  • Optimized for speed. Minimize memory allocations, compress data at the transport layer when possible. Groq, Mistral and OpenAI use brotli for HTTP compression instead of gzip, and POST's body to Google are gzip compressed.
  • Lean: Few dependencies. No unnecessary abstraction layer.

Scoreboard

Provider 🌐 Mode βž›In Outβž› Tool JSON Batch File Cite Text Probs Limits Usage Finish
alibaba πŸ‡¨πŸ‡³ Sync, Stream🧠 πŸ’¬πŸ“Έ πŸ’¬ βœ…πŸͺ¨πŸ•ΈοΈπŸ”€ ☁️ ❌ ❌ ❌ πŸŒ±πŸ“πŸ›‘ βœ… ❌ βœ… βœ…
anthropic πŸ‡ΊπŸ‡Έ Sync, Stream🧠 πŸ’¬πŸ“„πŸ“Έ πŸ’¬ βœ…πŸ•ΈοΈ πŸ“[] βœ… ❌ βœ… πŸ“πŸ›‘ ❌ βœ… βœ… βœ…
antigravity πŸ‡ΊπŸ‡Έ Sync, Stream🧠 πŸ’¬ πŸ’¬ ❌ πŸ“ ❌ ❌ ❌ ❌ ❌ ❌ βœ… βœ…
baseten πŸ‡ΊπŸ‡Έ Sync, Stream🧠 πŸ’¬πŸ“Έ πŸ’¬ βœ…πŸͺ¨πŸ”€ ☁️ ❌ ❌ ❌ πŸŒ±πŸ“πŸ›‘ ❌ ❌ βœ… βœ…
bfl πŸ‡©πŸ‡ͺ Sync πŸ’¬ πŸ“Έ ❌ ❌ βœ… ❌ ❌ 🌱 ❌ βœ… ❌ ❌
cerebras πŸ‡ΊπŸ‡Έ Sync, Stream🧠 πŸ’¬πŸ“Έ πŸ’¬ βœ…πŸͺ¨ βœ… ❌ ❌ ❌ πŸŒ±πŸ“πŸ›‘ βœ… βœ… βœ… βœ…
claudecode πŸ‡ΊπŸ‡Έ Sync, Stream🧠 πŸ’¬πŸ“„πŸ“Έ πŸ’¬ πŸ•ΈοΈ ❌ ❌ ❌ ❌ ❌ ❌ ❌ βœ… βœ…
cloudflare πŸ‡ΊπŸ‡Έ SystemOne, Sync, Stream🧠 πŸ’¬πŸ“Έ πŸŽ―πŸ’¬ πŸ’¨ βœ…[] ❌ ❌ ❌ πŸŒ±πŸ“ ❌ ❌ βœ… πŸ’¨
codex πŸ‡ΊπŸ‡Έ Sync, Stream🧠 πŸ’¬πŸ“Έ πŸ’¬ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ βœ… βœ…
cohere πŸ‡¨πŸ‡¦ Sync, Stream🧠 πŸ’¬ πŸ’¬ πŸ’¨πŸͺ¨ βœ… ❌ ❌ βœ… πŸŒ±πŸ“πŸ›‘ βœ… ❌ βœ… βœ…
deepseek πŸ‡¨πŸ‡³ Sync, Stream🧠 πŸ’¬ πŸ’¬ βœ…πŸͺ¨ ☁️ ❌ ❌ ❌ πŸ“πŸ›‘ βœ… ❌ βœ… βœ…
dsh πŸ‡¨πŸ‡³ Sync, Stream🧠 πŸ’¬πŸ“„ πŸ’¬ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ βœ…
gemini πŸ‡ΊπŸ‡Έ Embed, Realtime, Sync, Stream🧠 🎀πŸŽ₯πŸ’¬πŸ“„πŸ“Έ πŸŽ€πŸ’¬πŸ§¬ βœ…πŸͺ¨πŸ•ΈοΈπŸ”€ βœ…[] ❌ βœ… ❌ πŸŒ±πŸ›‘ ❌ ❌ βœ… βœ…
groq πŸ‡ΊπŸ‡Έ Sync, Stream🧠 πŸ’¬πŸ“Έ πŸ’¬ βœ…πŸͺ¨πŸ”€πŸ•ΈοΈ ☁️ ❌ ❌ ❌ πŸŒ±πŸ“πŸ›‘ ❌ βœ… βœ… βœ…
huggingface πŸ‡ΊπŸ‡Έ Sync, Stream🧠 πŸ’¬πŸ“Έ πŸ’¬ βœ…πŸͺ¨πŸ”€ βœ…[] ❌ ❌ ❌ πŸŒ±πŸ“πŸ›‘ βœ… βœ… βœ… βœ…
llamacpp 🏠 Embed, SystemOne, Realtime, Sync, Stream🧠 🎀πŸŽ₯πŸ’¬πŸ“Έ πŸŽ€πŸŽ―πŸ’¬πŸ§¬ βœ…πŸͺ¨πŸ”€ βœ…[] ❌ ❌ ❌ πŸŒ±πŸ“πŸ›‘ βœ… ❌ βœ… βœ…
mistral πŸ‡«πŸ‡· Sync, Stream🧠 πŸŽ€πŸ’¬πŸ“„πŸ“Έ πŸ’¬ βœ…πŸͺ¨ βœ…[] ❌ ❌ ❌ πŸŒ±πŸ“πŸ›‘ ❌ βœ… βœ… βœ…
ollama 🏠 Embed, SystemOne, Sync, Stream🧠 πŸ’¬πŸ“Έ πŸŽ―πŸ’¬πŸ§¬ βœ…πŸ”€ βœ…[] ❌ ❌ ❌ πŸŒ±πŸ“πŸ›‘ βœ… ❌ βœ… βœ…
openaichat πŸ‡ΊπŸ‡Έ Embed, Realtime, Sync, Stream🧠 πŸŽ€πŸ’¬πŸ“„πŸ“Έ πŸŽ€πŸ’¬πŸ“ΈπŸ§¬ βœ…πŸͺ¨ βœ… βœ… βœ… ❌ πŸŒ±πŸ“ βœ… βœ… βœ… βœ…
openairesponses πŸ‡ΊπŸ‡Έ Embed, Realtime, Sync, Stream🧠 πŸŽ€πŸ’¬πŸ“„πŸ“Έ πŸŽ€πŸ’¬πŸ“ΈπŸ§¬ βœ…πŸͺ¨πŸ•ΈοΈ βœ… βœ… ❌ ❌ ❌ ❌ βœ… βœ… βœ…
opencode πŸ‡ΊπŸ‡Έ Sync, Stream🧠 πŸ’¬πŸ“Έ πŸ’¬ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ βœ… βœ…
openrouter πŸ‡ΊπŸ‡Έ Sync, Stream🧠 πŸ’¬πŸ“Έ πŸ’¬ βœ…πŸ”€πŸͺ¨ ☁️[] ❌ ❌ ❌ πŸŒ±πŸ“πŸ›‘ ❌ ❌ βœ… βœ…
pi πŸ‡¦πŸ‡Ή Sync, Stream🧠 πŸ’¬πŸ“Έ πŸ’¬ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ βœ… βœ…
pollinations πŸ‡©πŸ‡ͺ Sync, Stream🧠 πŸ’¬πŸ“„πŸ“Έ πŸ’¬πŸ“Έ βœ…πŸͺ¨ ☁️ ❌ ❌ ❌ 🌱 ❌ ❌ βœ… βœ…
togetherai πŸ‡ΊπŸ‡Έ Sync, Stream🧠 πŸ’¬πŸ“Έ πŸ’¬πŸ“Έ βœ…πŸͺ¨ ☁️ ❌ ❌ ❌ πŸŒ±πŸ“πŸ›‘ ❌ ❌ βœ… βœ…
typesafe πŸ‡ΊπŸ‡Έ SystemOne πŸ’¬ 🎯 ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ βœ… ❌
xiaomi πŸ‡¨πŸ‡³ Sync, Stream🧠 🎀πŸŽ₯πŸ’¬πŸ“Έ πŸŽ€πŸ’¬ βœ…πŸͺ¨πŸ”€ ☁️ ❌ ❌ ❌ πŸ“πŸ›‘ ❌ ❌ βœ… βœ…
openaicompatible N/A Sync, Stream πŸ’¬ πŸ’¬ ❌ ❌ ❌ ❌ ❌ πŸ“πŸ›‘ ❌ ❌ βœ… βœ…
‼️ Click here for the legend of columns and symbols
  • 🏠: Runs locally.
  • Sync: Runs synchronously, the reply is only returned once completely generated
  • Stream: Streams the reply as it is generated. Occasionally less features are supported in this mode
  • 🧠: Has chain-of-thought thinking process
    • Both redacted (Anthropic, Gemini, OpenAI) and explicit (Deepseek R1, Qwen3, etc)
    • Many models can be used in both mode. In this case they will have two rows, one with thinking and one without. It is frequent that certain functionalities are limited in thinking mode, like tool calling.
  • βœ…: Implemented and works great
  • ❌: Not supported by genai. The provider may support it, but genai does not (yet). Please send a PR to add it!
  • πŸ’¬: Text
  • 🎯: Decision: typed answers returned by SystemOne
  • 🧬: Embedding: text vectors with qualified input order and semantic retrieval
  • Realtime: Bidirectional audio and text session through ConnectRealtime
  • WS: Measured over the WebSocket wire instead of HTTP.
  • πŸ“„: PDF: process a PDF as input, possibly with OCR
  • πŸ“Έ: Image: process an image as input; most providers support PNG, JPG, WEBP and non-animated GIF, or generate images
  • 🎀: Audio: process an audio file (e.g. MP3, WAV, Flac, Opus) as input, or generate audio
  • πŸŽ₯: Video: process a video (e.g. MP4) as input, or generate a video (e.g. Veo 3)
  • πŸ’¨: Feature is flaky (Tool calling) or inconsistent (Usage or Finish reason is not always reported)
  • 🌐: Country where the company is located
  • Tool: Tool calling, using genai.ToolDef; best is βœ…πŸͺ¨πŸ•ΈοΈπŸ”€ - πŸͺ¨: Tool calling can be forced; aka you can force the model to call a tool. This is great. - πŸ•ΈοΈ: Web search - πŸ”€: Supports messages and tool calls out of order
  • JSON: ability to output JSON in free form, or with a forced schema specified as a Go struct
    • βœ…: Supports both free form and with a schema
    • ☁️ :Supports only free form
      • πŸ“: Supports only a schema
      • []: Also supports an array at the root of the schema, most providers require an object
  • Batch: Process asynchronously batches during off peak hours at a discounts
  • Text: Text features
    • '🌱': Seed option for deterministic output
    • 'πŸ“': MaxTokens option to cap the amount of returned tokens
    • 'πŸ›‘': Stop sequence to stop generation when a token is generated
  • File: Upload and store large files via a separate API
  • Cite: Citation generation from a provided document, specially useful for RAG
  • Probs: Return logprobs to analyse each token probabilities
  • Limits: Returns the rate limits, including the remaining quota

Examples

The following examples intentionally use a variety of providers to show the extent at which you can pick and chose. Close each provider client when finished.

Text Basic βœ…

examples/txt_to_txt_sync/main.go: This selects a good default model based on Anthropic's currently published models, sends a prompt and prints the response as a string. πŸ’‘ Set ANTHROPIC_API_KEY.

func main() {
	ctx := context.Background()
	c, err := anthropic.New(ctx, genai.ModelGood)
	if c != nil {
		defer func() {
			if err := c.Close(); err != nil {
				log.Printf("Close: %v", err)
			}
		}()
	}
	if err != nil {
		log.Print(err)
		return
	}
	msgs := genai.Messages{
		genai.NewTextMessage("Give me a life advice that sounds good but is a bad idea in practice. Answer succinctly."),
	}
	result, err := c.GenSync(ctx, msgs)
	fmt.Println(result.String())
}

This may print:

"Follow your passion and the money will follow."

This ignores market realities, financial responsibilities, and the fact that passion alone doesn't guarantee income or career viability.

Multiple Text Completions

examples/txt_to_txt_sync_multi/main.go: This shows how to do multiple message round trips adding additional follow-up messages from users. Set OPENAI_API_KEY.

func main() {
	ctx := context.Background()
	c, err := anthropic.New(ctx, genai.ModelGood)
	if c != nil {
		defer func() {
			if err := c.Close(); err != nil {
				log.Printf("Close: %v", err)
			}
		}()
	}
	if err != nil {
		log.Print(err)
		return
	}
	msgs := genai.Messages{
		genai.NewTextMessage("Let's play a word association game. You pick a single word, then I pick the first word I think of, then you respond with a word, and so on.")
	}
	result, err := c.GenSync(ctx, msgs)
    if err != nil {
        panic(err)
    }
    // Show the message from ChatGPT
	fmt.Println(result.String())
    // Save the message in the collection of messages to build up context
    msgs = append(msgs, result.Message)
    // Add another user message
    msgs = append(msgs, genai.NewTextMessage("nightwish"))
    // Get another completion
    result, err := c.GenSync(ctx, msgs)
    // ...and so on.
}
Text Streaming 🏎

examples/txt_to_txt_stream/main.go: This is the same example as above, with the output streamed as it replies. This leverages go 1.23 iterators. Notice how little difference there is between both.

func main() {
	ctx := context.Background()
	c, err := anthropic.New(ctx, genai.ModelGood)
	if c != nil {
		defer func() {
			if err := c.Close(); err != nil {
				log.Printf("Close: %v", err)
			}
		}()
	}
	if err != nil {
		log.Print(err)
		return
	}
	msgs := genai.Messages{
		genai.NewTextMessage("Give me a life advice that sounds good but is a bad idea in practice."),
	}
	fragments, finish := c.GenStream(ctx, msgs)
	for f := range fragments {
		os.Stdout.WriteString(f.Text)
	}
	_, err = finish()
}
Text Thinking 🧠

examples/txt_to_txt_thinking/main.go: genai supports for implicit reasoning (e.g. Anthropic) and explicit reasoning (e.g. Deepseek). πŸ’‘ Set DEEPSEEK_API_KEY.

Snippet:

	c, err := deepseek.New(ctx, genai.ProviderOptionModel("deepseek-flash"), genai.EffortHigh, genai.IncludeReasoning)
	if c != nil {
		defer func() {
			if err := c.Close(); err != nil {
				log.Printf("Close: %v", err)
			}
		}()
	}
	if err != nil {
		log.Print(err)
		return
	}
	msgs := genai.Messages{
		genai.NewTextMessage("Give me a life advice that sounds good but is a bad idea in practice."),
	}
	fragments, finish := c.GenStream(ctx, msgs)
	for f := range fragments {
		if f.Reasoning != "" {
			// ...
		} else if f.Text != "" {
			// ...
		}
	}
Text Citations ✍

examples/txt_to_txt_citations/main.go: Send entire documents and leverage providers which support automatic citations (Cohere, Anthropic) to leverage their functionality for a supercharged RAG. πŸ’‘ Set COHERE_API_KEY.

Snippet:

	const context = `...` // Introduction of On the Origin of Species by Charles Darwin...
	msgs := genai.Messages{{
		Requests: []genai.Request{
			{
				Doc: genai.Doc{
					Filename: "On-the-Origin-of-Species-by-Charles-Darwin.txt",
					Src:      strings.NewReader(context),
				},
			},
			{Text: "When did Darwin arrive home?"},
		},
	}}
	res, _ := c.GenSync(ctx, msgs)
	for _, r := range res.Replies {
		if !r.Citation.IsZero() {
			fmt.Printf("Citation:\n")
			for _, src := range r.Citation.Sources {
				fmt.Printf("- %q\n", src.Snippet)
			}
		}
	}
	fmt.Printf("\nAnswer: %s\n", res.String())

When asked When did Darwin arrive home? with the introduction of On the Origin of Species by Charles Darwin passed in as a document, this may print:

Citation:

  • "excerpt from Charles Darwin's work 'On the Origin of Species'"
  • "returned home in 1837."

Answer: 1837 was when Darwin returned home and began to reflect on the facts he had gathered during his time on H.M.S. Beagle.

Text Websearch πŸ•ΈοΈ

examples/txt_to_txt_websearch-sync/main.go: Searches the web to answer your question. πŸ’‘ Set GEMINI_API_KEY.

Snippet:

	c, err := gemini.New(ctx, genai.ModelCheap)
	if c != nil {
		defer func() {
			if err := c.Close(); err != nil {
				log.Printf("Close: %v", err)
			}
		}()
	}
	if err != nil {
		log.Print(err)
		return
	}
	msgs := genai.Messages{{
		Requests: []genai.Request{
			{Text: "Who holds ultimate power of Canada? Answer succinctly."},
		},
	}}

	opts := genai.GenOptionWeb{Search: true, Fetch: true}
	res, _ := c.GenSync(ctx, msgs, &opts)
	for _, r := range res.Replies {
		if !r.Citation.IsZero() {
			fmt.Printf("Sources:\n")
			for _, src := range r.Citation.Sources {
				switch src.Type {
				case genai.CitationWeb:
					fmt.Printf("- %s / %s\n", src.Title, src.URL)
				case genai.CitationWebImage:
					fmt.Printf("- image: %s\n", src.URL)
				}
			}
		}
	}
	fmt.Printf("\nAnswer: %s\n", res.String())

Try it locally:

go run github.com/maruel/genai/examples/txt_to_txt_websearch-sync@latest

When asked Who holds ultimate power of Canada?, this may print:

Sources:

(...)

Image: https://learn.parl.ca/understanding-comprendre/images/articles/monarch-and-governor-general/house-of-commons.jpg

(...)

Answer: The ultimate power in Canada constitutionally resides with the monarch (King Charles III) as the head of state, with executive authority formally vested in him. However, (...)

Text Websearch (streaming) πŸ”οΈ

examples/txt_to_txt_websearch-stream/main.go: Searches the web to answer your question and streams the output to the console. πŸ’‘ Set GEMINI_API_KEY.

go run github.com/maruel/genai/examples/txt_to_txt_websearch-stream@latest

Same as above, but streaming.

Log probabilities

examples/txt_to_txt_logprobs/main.go: List the alternative tokens that were considered during generation. This helps tune Temperature, TopP or TopK.

Try it locally:

go run github.com/maruel/genai/examples/txt_to_txt_logprobs@latest

When asked Tell a joke, this may print:

Provider huggingface
  Reply:
    Why don't scientists trust atoms?

    Because they make up everything!
  Logprobs:
    *    -0.000082: "Why"
         -9.625082: "Here"
        -11.250082: "What"
        -13.875082: "A"
        -14.500082: "How"
    *    -0.000003: " don"
        -14.125003: " do"
        -14.625003: " did"
        -14.625003: " dont"
        -14.875003: " didn"
    *    -0.000001: "'t"
        -14.000001: "’t"
        -18.062500: "'"
        -18.875000: "'T"
        -19.812500: "'s"
    *    -0.000002: " scientists"
        -14.250002: " Scientists"
        -14.250002: " eggs"
        -15.125002: " skeletons"
        -16.125002: " programmers"
    *    -0.000000: " trust"
        -16.250000: " trusts"
        -16.250000: " Trust"
        -17.250000: " like"
        -18.000000: " trusted"
    *    -0.000006: " atoms"
        -13.250006: "atoms"
        -13.500006: " stairs"
        -14.625006: " their"
        -15.000006: " electrons"
    *    -0.000011: "?\n\n"
        -12.125011: "?\n"
        -12.125011: "?"
        -14.750011: "?\n\n"
        -16.500011: " anymore"
(...)
Text Tools 🧰

examples/txt_to_txt_tool-sync/main.go: A LLM can both retrieve information and act on its environment through tool calling. This unblocks a whole realm of possibilities. Our design enables dense strongly typed code that favorably compares to python. πŸ’‘ Set CEREBRAS_API_KEY.

Snippet:

	type numbers struct {
		A int `json:"a"`
		B int `json:"b"`
	}
	msgs := genai.Messages{
		genai.NewTextMessage("What is 3214 + 5632? Call the tool \"add\" to tell me the answer. Do not explain. Be terse. Include only the answer."),
	}
	opts := genai.GenOptionTools{
		Tools: []genai.ToolDef{
			{
				Name:        "add",
				Description: "Add two numbers together and provides the result",
				Callback: func(ctx context.Context, input *numbers) (string, error) {
					return fmt.Sprintf("%d", input.A+input.B), nil
				},
			},
		},
		// Force the LLM to do a tool call.
		Force: genai.ToolCallRequired,
	}

	// Run the loop.
	res, _, _ := adapters.GenSyncWithToolCallLoop(ctx, c, msgs, &opts)
	// Print the answer which is the last message generated.
	fmt.Println(res[len(res)-1].String())

When asked What is 3214 + 5632?, this may print:

8846

Text Tools (streaming) 🐝

examples/txt_to_txt_tool-stream/main.go: Leverage a thinking model to see the thinking process while trying to use tool calls to answer the user's question. This enables keeping the user updated to see the progress. πŸ’‘ Set GROQ_API_KEY.

Snippet:

	fragments, finish := adapters.GenStreamWithToolCallLoop(ctx, p, msgs, &opts)
	for f := range fragments {
		if f.Reasoning != "" {
			// ...
		} else if f.Text != "" {
			// ...
		} else if !f.ToolCall.IsZero() {
			// ...
		}
	}

``

When asked What is 3214 + 5632?, this may print:

# Reasoning

User wants result of 3214+5632 using tool "add". Must be terse, only answer, no explanation. Need to call add function with a=3214, b=5632.

# Tool call

{fc_e9b9677b-898c-46df-9deb-39122bd6c69a add {"a":3214,"b":5632} map[] {}}

# Answer

8846

Text Tools (manual)

examples/txt_to_txt_tool-manual/main.go: Runs a manual loop and runs tool calls directly. πŸ’‘ Set CEREBRAS_API_KEY.

Snippet:

	res, _ := c.GenSync(ctx, msgs, &opts)
	// Add the assistant's message to the messages list.
	msgs = append(msgs, res.Message)
	// Process the tool call from the assistant.
	msg, _ := res.DoToolCalls(ctx, opts.Tools)
	// Add the tool call response to the messages list.
	msgs = append(msgs, msg)
	// Follow up so the LLM can interpret the tool call response.
	res, _ = c.GenSync(ctx, msgs, &opts)
Realtime Tools 🎀🧰

adapters/example_test.go: RealtimeWithToolCallLoop executes tool calls from a realtime session and sends the results back. It follows tool changes sent with RealtimeUpdate.

Snippet:

	tools := []genai.ToolDef{{
		Name:        "get_today_clock_time",
		Description: "Get the current clock time and today's date.",
		Callback: func(context.Context, *struct{}) (string, error) {
			return time.Now().Format("Monday 2006-01-02 15:04:05"), nil
		},
	}}
	s, err := c.ConnectRealtime(ctx, &genai.RealtimeOptionTools{Tools: tools}, genai.RealtimeTurnManual)
	if err != nil {
		log.Fatal(err)
	}
	s = adapters.RealtimeWithToolCallLoop(s, tools)
	defer s.Close()
	if err := s.Send(ctx, genai.RealtimeInputContent{Text: "What time is it?", Generate: true}); err != nil {
		log.Fatal(err)
	}
	for e, err := range s.Receive(ctx) {
		if err != nil {
			log.Fatal(err)
		}
		switch e.Kind {
		case genai.RealtimeEventContentText:
			fmt.Print(e.Content.Text)
		case genai.RealtimeEventTurnDone:
			return
		}
	}
Text Decode reply as a struct βš™

examples/txt_to_txt_decode-json/main.go: Tell the LLM to use a specific Go struct to determine the JSON schema to generate the response. This is much more lightweight than tool calling!

It is very useful when we want the LLM to make a choice between values, to return a number or a boolean (true/false). Enums are supported. πŸ’‘ Set OPENAI_API_KEY.

Snippet:

	msgs := genai.Messages{
		genai.NewTextMessage("Is a circle round? Reply as JSON."),
	}
	var circle struct {
		Round bool `json:"round"`
	}
	opts := genai.GenOptionText{DecodeAs: &circle}
	res, _ := c.GenSync(ctx, msgs, &opts)
	res.Decode(&circle)
	fmt.Printf("Round: %v\n", circle.Round)

This will print:

Round: true

Text to Typed Decisions ❓

Ask yes/no, choice, or score questions about text or JSON state with Provider.SystemOne(). Set named questions in genai.Questions. Read typed answers under the same names from genai.Answers.

	questions := genai.Questions{
		"billing": {Type: genai.QuestionNoul, Instructions: genai.Text("Is this about billing?")},
	}
	res, err := c.SystemOne(ctx, &genai.SystemOneRequest{
		State: genai.Text("I was charged twice."), Questions: questions,
	})
	if err != nil {
		return err
	}
	fmt.Println(res.Answers["billing"].Noul)

A single inline JSON Doc supplies the state when State is unset. See the TypeSafe, Cloudflare, and llama.cpp for setup and constraints.

Complete examples: hosted text decisions, local text decisions, and local image and text decisions.

Text to Image πŸ“Έ

examples/txt_to_img/main.go: Use Together.AI's free (!) image generation albeit with low rate limit.

Some providers return an URL that must be fetched manually within a few minutes or hours, some return the data inline. This example handles both cases. πŸ’‘ Set TOGETHER_API_KEY.

Snippet:

	msgs := genai.Messages{
		genai.NewTextMessage("Carton drawing of a husky playing on the beach."),
	}
	result, _ := c.GenSync(ctx, msgs)
	for _, r := range result.Replies {
		if r.Doc.IsZero() {
			continue
		}
		// The image can be returned as an URL or inline, depending on the provider.
		var src io.Reader
		if r.Doc.URL != "" {
			req, _ := c.HTTPClient().Get(r.Doc.URL)
			src = req.Body
			defer req.Body.Close()
		} else {
			src = r.Doc.Src
		}
		b, _ := io.ReadAll(src)
		os.WriteFile(r.Doc.GetFilename(), b, 0o644)
	}

Try it locally:

go run github.com/maruel/genai/examples/txt_to_img@latest

This may generate:

content.jpg

This generated picture shows a fake signature. I decided to keep this example as a reminder that the result comes from the data harvested that was created by real humans.

Image-Text to Video πŸŽ₯

examples/img-txt_to_vid/main.go: Leverage the content.jpg file generated in txt_to_img example to ask Veo 3 from Google to generate a video based on the image. πŸ’‘ Set GEMINI_API_KEY.

Snippet:

	// Warning: this is expensive.
	c, err := gemini.New(ctx, genai.ProviderOptionModel("veo-3.0-fast-generate-preview"))
	if c != nil {
		defer func() {
			if err := c.Close(); err != nil {
				log.Printf("Close: %v", err)
			}
		}()
	}
	if err != nil {
		log.Print(err)
		return
	}
	f, _ := os.Open("content.jpg")
	defer f.Close()
	msgs := genai.Messages{
		genai.Message{Requests: []genai.Request{
			{Text: "Carton drawing of a husky playing on the beach."},
			{Doc: genai.Doc{Src: f}},
		}},
	}
	res, _ := c.GenSync(ctx, msgs)
	// Save the file in Replies like in the previous example ...

Try it locally:

go run github.com/maruel/genai/examples/img-txt_to_vid@latest

This may generate:

content.avif

⚠ The MP4 has been recompressed to AVIF via compress.sh so GitHub can render it. The drawback is that audio is lost. View the original MP4 with audio (!) at content.mp4. May not work on Safari.

This is very impressive, but also very expensive.

Image-Text to Image πŸ–Œ

examples/img-txt_to_img/main.go: Edit an image with a prompt. Leverage the content.jpg file generated in txt_to_img example. πŸ’‘ Set BFL_API_KEY.

go run github.com/maruel/genai/examples/img-txt_to_img@latest

This may generate:

content2.jpg

Image-Text to Image-Text 🍌

examples/img-txt_to_img-txt/main.go: Leverage the content.jpg file generated in txt_to_img example to ask gemini-2.5-flash-image-preview to change the image with a prompt and ask the model to explain what it did. πŸ’‘ Set GEMINI_API_KEY.

Snippet:

	// Warning: This is a bit expensive.
	c, err := gemini.New(ctx,
		genai.ProviderOptionModel("gemini-2.5-flash-image-preview"),
		genai.ProviderOptionModalities{genai.ModalityImage, genai.ModalityText},
	)
	if c != nil {
		defer func() {
			if err := c.Close(); err != nil {
				log.Printf("Close: %v", err)
			}
		}()
	}
	if err != nil {
		log.Print(err)
		return
	}
	// ...
	res, _ := c.GenSync(ctx, msgs, &gemini.GenOption{ReasoningBudget: 0})

Try it locally:

go run github.com/maruel/genai/examples/img-txt_to_img-txt@latest

This may generate:

Of course! Here's an updated image with more animals. I added a playful dolphin jumping out of the water and a flock of seagulls flying overhead. I chose these animals to enhance the beach scene and create a more dynamic and lively atmosphere.

Wrote: content.png

content.png

This is quite impressive, but also quite expensive.

Image-Text to Text πŸ‘

examples/img-txt_to_txt/main.go: Run vision to analyze a picture provided as an URL (source: wikipedia). The response is streamed out the console as the reply is generated. πŸ’‘ Set MISTRAL_API_KEY.

go run github.com/maruel/genai/examples/img-txt_to_txt@latest

This may generate:

The image depicts a single ripe banana. It has a bright yellow peel with a few small brown spots, indicating ripeness. The banana is curved, which is typical of its natural shape, and it has a stem at the top. The overall appearance suggests that it is ready to be eaten.

Image-Text to Text (local) 🏠

examples/img-txt_to_txt_local/main.go: is very similar to the previous example!

Use cmd/llama-serve to run a LLM locally, including tool calling and vision!

Start llama-server locally either by yourself or with this utility:

go run github.com/maruel/genai/cmd/llama-serve@latest \
  -model ggml-org/gemma-3-4b-it-GGUF/gemma-3-4b-it-Q8_0.gguf -- \
  --temp 1.0 --top-p 0.95 --top-k 64 \
  --jinja -fa -c 0 --no-warmup

Run vision 100% locally on CPU with only 8GB of RAM. No GPU required!

go run github.com/maruel/genai/examples/img-txt_to_txt_local@latest
Video-Text to Text 🎞️

examples/vid-txt_to_txt/main.go: Run vision to analyze a video. πŸ’‘ Set GEMINI_API_KEY.

Using this video:

video.avif

Try it locally:

go run github.com/maruel/genai/examples/vid-txt_to_txt@latest

When asked What is the word, this generates:

Banana

Audio-Text to Text 🎀

examples/aud-txt_to_txt/main.go: Analyze an audio file. πŸ’‘ Set OPENAI_API_KEY.

Try it locally:

go run github.com/maruel/genai/examples/vid-txt_to_txt@latest

When asked What was the word?, this generates:

The word was "orange."

Usage and Quota πŸŸπŸ§€πŸ₯£

examples/txt_to_txt_quota/main.go: Prints the tokens processed and generated for the request and the remaining quota if the provider supports it. πŸ’‘ Set GROQ_API_KEY.

Snippet:

	msgs := genai.Messages{
		genai.NewTextMessage("Describe poutine as a French person who just arrived in QuΓ©bec"),
	}
	res, _ := c.GenSync(ctx, msgs)
	fmt.Println(res.String())
	fmt.Printf("\nTokens usage: %s\n", res.Usage.String())

This may generate:

Β«β€―Je viens tout juste d’arriver au QuΓ©bec et, pour Γͺtre honnΓͺte, je n’avais jamais entendu parler du fameux Β«β€―poutineβ€―Β» avant de mettre le pied dans un petit resto du coin.β€―Β»

(...)

Tokens usage: in: 83 (cached 0), reasoning: 0, out: 818, total: 901, requests/2025-08-29 15:58:13: 499999/500000, tokens/2025-08-29 15:58:12: 249916/250000

In addition to the token usage, remaining quota is printed.

Text with any provider ⁉

examples/txt_to_txt_any/main.go: Let the user chose the provider by name.

The relevant environment variable (e.g. ANTHROPIC_API_KEY, OPENAI_API_KEY, etc) is used automatically for authentication.

Automatically selects a models on behalf of the user. Wraps the explicit thinking tokens if needed.

Supports ollama and llama-server even if they run on a remote host or non-default port.

Snippet:

	names := strings.Join(slices.Sorted(maps.Keys(providers.Available(ctx))), ", ")
	provider := flag.String("provider", "", "provider to use, "+names)
	flag.Parse()

	cfg := providers.All[*provider]
	c, err := cfg.Factory(ctx, genai.ModelGood)
	if c != nil {
		defer func() {
			if err := c.Close(); err != nil {
				log.Printf("Close: %v", err)
			}
		}()
	}
	if err != nil {
		log.Print(err)
		return
	}
	p := adapters.WrapReasoning(c)
	res, _ := p.GenSync(...)

Try it locally:

go run github.com/maruel/genai/examples/txt_to_txt_any@latest \
    -provider cerebras \
    "Tell a good sounding advice that is a bad idea in practice."
OAuth2 Authentication πŸ”‘

Two examples demonstrate full OAuth2 browser login flows with PKCE:

examples/txt_to_txt_oauth2_codex/main.go: Authenticate via OpenAI's auth server (same flow as Codex CLI) and talk to the ChatGPT backend API. Tokens are cached and refreshed automatically. No configuration needed.

go run github.com/maruel/genai/examples/txt_to_txt_oauth2_codex@latest

examples/txt_to_txt_oauth2_gemini/main.go: Authenticate via Google OAuth2 and talk to the Gemini API. Requires a Google Cloud project with OAuth2 Desktop credentials.

go run github.com/maruel/genai/examples/txt_to_txt_oauth2_gemini@latest -client-id=ID -client-secret=SECRET

Embeddings

genai.Provider.Embed supports Gemini, llama.cpp, Ollama, OpenAI Chat and OpenAI Responses clients. Other providers return base.ErrNotSupported. Configure an explicit embedding model with genai.ProviderOptionModel; automatic generation model selection does not choose an embedding model.

c, err := gemini.New(ctx, genai.ProviderOptionModel("gemini-embedding-2"))
if err != nil {
    return err
}
// Close c when finished.
out, err := c.Embed(ctx, &genai.EmbeddingRequest{
    Inputs: []genai.Request{{Text: "A kitten plays with yarn."}, {Text: "A cat plays with string."}},
})
if err != nil {
    return err
}
// out.Embeddings[i] corresponds to the ith input request.

See the text, image and audio embedding example.

Model Selection

For automatic model selection, pass one of the marker constants directly:

// Use genai.ModelCheap for the cheapest model, genai.ModelGood for a good
// everyday model, or genai.ModelSOTA for the state-of-the-art model.
c, err := anthropic.New(ctx, genai.ModelGood)
if c != nil {
	defer func() {
		if err := c.Close(); err != nil {
			log.Printf("Close: %v", err)
		}
	}()
}
if err != nil {
	log.Print(err)
	return
}

For a specific model, wrap the model ID with ProviderOptionModel:

// Specific model selection
c, err := anthropic.New(ctx, genai.ProviderOptionModel("claude-sonnet-4-20250514"))
if c != nil {
	defer func() {
		if err := c.Close(); err != nil {
			log.Printf("Close: %v", err)
		}
	}()
}
if err != nil {
	log.Print(err)
	return
}

For Gemini, the same option selects a Gemini model:

c, err := gemini.New(ctx, genai.ProviderOptionModel("gemini-2.5-flash"))
if c != nil {
	defer func() {
		if err := c.Close(); err != nil {
			log.Printf("Close: %v", err)
		}
	}()
}
if err != nil {
	log.Print(err)
	return
}

Models πŸ—’

Snapshot of all the supported models at docs/MODELS.md is updated weekly.

Try it locally:

go install github.com/maruel/genai/cmd/...@latest

list-models -provider huggingface

Providers with free tier πŸ’Έ

As of August 2025, the following services offer a free tier (other limits apply):

TODO

PRs are appreciated for any of the following. No need to ask! Just send a PR and make it pass CI checks. ❀️

Features
Providers

I'd be delighted if you want to contribute any missing provider being added, I'm particularly looking forward to these:

I'm also looking to disconnect more the scoreboard from the Go code. I believe the scoreboard is useful in itself and is not Go specific. I appreciate ideas towards achieving this, send them my way!

Thanks in advance! πŸ™

Made with ❀️ by Marc-Antoine Ruel

Documentation ΒΆ

Overview ΒΆ

Package genai is the opiniated high performance professional-grade AI package for Go.

It provides a generic interface to interact with various LLM providers, while allowing full access to each provider's full capabilities.

Check out the examples for a quick start.

Example (GenSyncWithToolCallLoop_with_custom_HTTP_Header) ΒΆ
package main

import (
	"context"
	"fmt"
	"log"
	"net/http"
	"time"

	"github.com/maruel/roundtrippers"

	"github.com/maruel/genai"
	"github.com/maruel/genai/adapters"
	"github.com/maruel/genai/providers/anthropic"
)

func main() {
	// Modified version of the example in package adapters, with a custom header.
	//
	// As of June 2025, interleaved thinking can be enabled with a custom header.
	// https://docs.anthropic.com/en/docs/build-with-claude/extended-thinking#interleaved-thinking
	wrapper := func(h http.RoundTripper) http.RoundTripper {
		return &roundtrippers.Header{
			Transport: h,
			Header:    http.Header{"anthropic-beta": []string{"interleaved-thinking-2025-05-14"}},
		}
	}
	ctx := context.Background()
	c, err := anthropic.New(ctx, genai.ProviderOptionModel("claude-sonnet-4-20250514"), genai.ProviderOptionTransportWrapper(wrapper))
	if c != nil {
		defer func() {
			if err := c.Close(); err != nil {
				log.Printf("Close: %v", err)
			}
		}()
	}
	if err != nil {
		log.Print(err)
		return
	}
	msgs := genai.Messages{genai.NewTextMessage("What season is MontrΓ©al currently in?")}
	opts := genai.GenOptionTools{
		Tools: []genai.ToolDef{locationClockTime},
		// Force the LLM to do a tool call first.
		Force: genai.ToolCallRequired,
	}
	newMsgs, _, err := adapters.GenSyncWithToolCallLoop(ctx, c, msgs, &opts)
	if err != nil {
		log.Print(err)
		return
	}
	fmt.Printf("%s\n", newMsgs[len(newMsgs)-1].String())
}

var locationClockTime = genai.ToolDef{
	Name:        "get_today_date_current_clock_time",
	Description: "Get the current clock time and today's date.",
	Callback: func(ctx context.Context, e *location) (string, error) {
		if e.Location != "MontrΓ©al" {
			return "ask again with MontrΓ©al", nil
		}
		return time.Now().Format("Monday 2006-01-02 15:04:05"), nil
	},
}

type location struct {
	Location string `json:"location" json_description:"Location to ask the current time in"`
}

Index ΒΆ

Examples ΒΆ

Constants ΒΆ

View Source
const (
	// ModalityAudio is support for audio formats like MP3, WAV, Opus, Flac, etc.
	ModalityAudio = scoreboard.ModalityAudio
	// ModalityDecision is typed answers returned by SystemOne.
	ModalityDecision = scoreboard.ModalityDecision
	// ModalityDocument is support for PDF with multi-modal comprehension, both images and text. This includes
	// code blocks.
	ModalityDocument = scoreboard.ModalityDocument
	// ModalityEmbedding is a vector returned by Embed.
	ModalityEmbedding = scoreboard.ModalityEmbedding
	// ModalityImage is support for image formats like PNG, JPEG, often single frame GIF, and WEBP.
	ModalityImage = scoreboard.ModalityImage
	// ModalityText is for raw text.
	ModalityText = scoreboard.ModalityText
	// ModalityVideo is support for video formats like MP4 or MKV.
	ModalityVideo = scoreboard.ModalityVideo
)

Variables ΒΆ

This section is empty.

Functions ΒΆ

This section is empty.

Types ΒΆ

type Answer ΒΆ added in v0.10.0

type Answer struct {
	// Type is the type of the question this is the answer to.
	Type QuestionType `json:"type"`
	// Noul is the probability that the answer is yes, from 0 to 1. It is set for QuestionNoul.
	Noul float64 `json:"noul,omitzero"`
	// Choice is the highest probability option. It is set for QuestionChoice.
	Choice string `json:"choice,omitzero"`
	// Score is the probability weighted value across the levels. It can land between levels. It is set for
	// QuestionScore.
	Score float64 `json:"score,omitzero"`
	// Confidence is how certain the model is, derived from Probabilities. It is set for QuestionChoice and
	// QuestionScore.
	Confidence float64 `json:"confidence,omitzero"`
	// Probabilities maps each option, or each level as a string key, to its probability. It is set for
	// QuestionChoice and QuestionScore.
	Probabilities map[string]float64 `json:"probabilities,omitzero"`
	// Legend maps each level as a string key to the description passed in Question.Score for that level.
	// It is set for QuestionScore.
	Legend ScoreLegend `json:"legend,omitzero"`
	// contains filtered or unexported fields
}

Answer is the answer to a Question.

It is a union discriminated by Type. Only the fields that match Type are set, and MarshalJSON writes only those fields, with the shape the API uses, so a probability or a score of 0 is not dropped.

An answer type the server adds later is kept as is instead of making the whole reply unusable.

func (*Answer) MarshalJSON ΒΆ added in v0.10.0

func (a *Answer) MarshalJSON() ([]byte, error)

MarshalJSON implements json.Marshaler.

The fields of the union that do not match Type are omitted, even when they are zero, so that a score of 0 or a probability of 0 is preserved.

func (*Answer) UnmarshalJSON ΒΆ added in v0.10.0

func (a *Answer) UnmarshalJSON(b []byte) error

UnmarshalJSON implements json.Unmarshaler.

type Answers ΒΆ added in v0.10.0

type Answers map[string]*Answer

Answers holds the answer to each question, keyed by the question name. Entries must not be nil.

type Array ΒΆ added in v0.10.0

type Array []any

Array is DecisionContent that is a JSON array.

The values are any, like Object: they must be JSON encodable. They are not restricted to DecisionContent items.

func (Array) Validate ΒΆ added in v0.10.0

func (a Array) Validate() error

Validate implements internal.Validatable.

Every item is validated so that the error names the offending index.

type AudioFormat ΒΆ added in v0.11.0

type AudioFormat struct {
	SampleRate int
}

AudioFormat describes raw mono PCM16LE at SampleRate Hz. genai never resamples; callers convert at the capture and playback boundaries.

func (*AudioFormat) Validate ΒΆ added in v0.11.0

func (f *AudioFormat) Validate() error

Validate checks that SampleRate is positive.

type CacheEntry ΒΆ

type CacheEntry interface {
	GetID() string
	GetDisplayName() string
	GetExpiry() time.Time
}

CacheEntry is one file (or GenSync request) cached on the provider for reuse.

type Citation ΒΆ

type Citation struct {
	// CitedText is the text that was cited.
	CitedText string `json:"cited_text,omitzero"`
	// StartIndex is the starting character position of the citation in the answer (0-based).
	StartIndex int64 `json:"start_index,omitzero"`
	// EndIndex is the ending character position of the citation in the answer (0-based, exclusive).
	EndIndex int64 `json:"end_index,omitzero"`

	// Sources contains information about the source documents or tools that support this citation.
	Sources []CitationSource `json:"sources,omitzero"`
	// contains filtered or unexported fields
}

Citation represents a reference to source material that supports content. It provides a unified interface for different provider citation formats.

Normally one of CitedText or StartIndex/EndIndex is set.

func (*Citation) IsZero ΒΆ

func (c *Citation) IsZero() bool

IsZero returns true if the citation is empty.

func (*Citation) Validate ΒΆ

func (c *Citation) Validate() error

Validate ensures the citation is valid.

type CitationSource ΒΆ

type CitationSource struct {
	// Type indicates the source type.
	Type CitationType `json:"type,omitzero"`
	// ID is a unique identifier for the source (e.g., document ID, tool call ID).
	ID string `json:"id,omitzero"`
	// Title is the human-readable title of the source.
	Title string `json:"title,omitzero"`
	// URL is the web URL for the source, if applicable.
	URL string `json:"url,omitzero"`
	// Snippet is a snippet from the source, if applicable. It is the web search query for CitationWebQuery.
	Snippet string `json:"snippet,omitzero"`
	// StartCharIndex is the starting character position of the citation in the sourced document (0-based).
	StartCharIndex int64 `json:"start_index,omitzero"`
	// EndCharIndex is the ending character position of the citation in the sourced document (0-based, exclusive).
	EndCharIndex int64 `json:"end_index,omitzero"`
	// StartPageNumber is the starting page number of the citation in the sourced document (1-based).
	StartPageNumber int64 `json:"start_page_number,omitzero"`
	EndPageNumber   int64 `json:"end_page_number,omitzero"`
	// StartBlockIndex is the starting block index of the citation in the sourced document (0-based).
	StartBlockIndex int64 `json:"start_block_index,omitzero"`
	EndBlockIndex   int64 `json:"end_block_index,omitzero"`

	// Date is the date of the source, if applicable.
	Date string `json:"date,omitzero"`
	// Metadata contains additional source-specific information.
	// For document sources: document index, page numbers, etc.
	// For tool sources: tool output, function name, etc.
	// For web sources: encrypted index, search result info, etc.
	Metadata map[string]any `json:"metadata,omitzero"`
	// contains filtered or unexported fields
}

CitationSource represents a source that supports a citation.

func (*CitationSource) IsZero ΒΆ

func (cs *CitationSource) IsZero() bool

IsZero returns true if the citation source is empty.

func (*CitationSource) Validate ΒΆ

func (cs *CitationSource) Validate() error

Validate ensures the citation source is valid.

type CitationType ΒΆ

type CitationType int32

CitationType is a citation that a model returned as part of its reply.

const (
	// CitationWebQuery is an query used as part of a web search.
	CitationWebQuery CitationType = iota + 1
	// CitationWeb is an URL from a web search.
	CitationWeb
	// CitationWebImage is an URL to an image from a web search.
	CitationWebImage
	// CitationDocument is from a document provided as input or explicitly referenced.
	CitationDocument
	// CitationTool is when the provider refers to the result of a tool call in its answer.
	CitationTool
)

type DecisionContent ΒΆ added in v0.10.0

type DecisionContent interface {
	// Validate ensures the content is valid.
	Validate() error
	// contains filtered or unexported methods
}

DecisionContent is text, a JSON object, or a JSON array.

It is implemented by Text, Object and Array, and by JSON state read from a Doc. It marshals to the value itself, not to an object with fields. Use nil when the field is optional and is left unset.

It is used for state, instructions and criteria descriptions. It is meant to be sent; the API only returns it in Answer.Legend, decoded back by ScoreLegend, since an interface cannot be decoded on its own.

func ParseDecisionContent ΒΆ added in v0.10.0

func ParseDecisionContent(b []byte) (DecisionContent, error)

ParseDecisionContent decodes one JSON value into its DecisionContent variant.

The variant is picked from the JSON token, not from the Go value a decoder would produce for an any, so what it accepts is unambiguous.

type DecisionUsage ΒΆ added in v0.10.0

type DecisionUsage struct {
	// InputTokens counts the prompt tokens evaluated for all questions.
	InputTokens int64 `json:"input_tokens"`
	// OutputTokens counts output tokens reported by the provider, often zero for decision models.
	OutputTokens int64 `json:"output_tokens"`
	// ReasoningTokens counts reasoning tokens within OutputTokens, when reported by the provider.
	ReasoningTokens int64 `json:"reasoning_tokens,omitzero"`
}

DecisionUsage reports the token usage of a request.

func (*DecisionUsage) Validate ΒΆ added in v0.10.0

func (u *DecisionUsage) Validate() error

Validate rejects negative token counts.

type Doc ΒΆ

type Doc struct {
	// Filename is the name of the file. For many providers, only the extension
	// is relevant. They only use mime-type, which is derived from the filename's
	// extension. When an URL is provided or when the object provided to Document
	// implements a method with the signature `Name() string`, like an
	// `*os.File`, Filename is optional.
	Filename string `json:"filename,omitzero"`
	// Src is raw document data. It is perfectly fine to use a bytes.NewReader() or *os.File.
	Src io.ReadSeeker `json:"bytes,omitzero"`
	// URL is the reference to the raw data. When set, the mime-type is derived from the URL.
	URL string `json:"url,omitzero"`
	// contains filtered or unexported fields
}

Doc is a document.

func (*Doc) GetFilename ΒΆ

func (d *Doc) GetFilename() string

GetFilename returns the filename to use for the document, querying the Document's name if available.

func (*Doc) IsZero ΒΆ

func (d *Doc) IsZero() bool

IsZero returns true if the document is empty.

func (*Doc) MIMEType ΒΆ added in v0.11.0

func (d *Doc) MIMEType() string

MIMEType returns the mime type the filename or URL declares, or an empty string when the extension is unknown.

func (*Doc) MarshalJSON ΒΆ

func (d *Doc) MarshalJSON() ([]byte, error)

MarshalJSON implements the json.Marshaler interface.

func (*Doc) Read ΒΆ

func (d *Doc) Read(maxSize int64) (string, []byte, error)

Read reads the document content into memory.

maxSize must be positive. Oversized content is rejected, including unseekable sources and content that grows after its size is checked. It returns the mime type, the raw bytes and an error if any.

func (*Doc) UnmarshalJSON ΒΆ

func (d *Doc) UnmarshalJSON(b []byte) error

UnmarshalJSON implements the json.Unmarshaler interface.

func (*Doc) Validate ΒΆ

func (d *Doc) Validate() error

Validate ensures the block is valid.

type EffortLevel ΒΆ added in v0.11.0

type EffortLevel string

EffortLevel controls how much effort a model spends reasoning about its response. Providers map these levels to their native controls; the token budget and latency of a level depend on the provider and model. Empty preserves the provider's default.

const (
	// EffortOff disables thinking where possible, or selects the lowest supported effort.
	EffortOff EffortLevel = "off"
	// EffortMinimal requests the smallest supported reasoning effort without disabling thinking.
	// Providers use native minimal effort where supported, or their lowest enabled level.
	EffortMinimal EffortLevel = "minimal"
	// EffortLow favors lower cost and latency.
	EffortLow EffortLevel = "low"
	// EffortMedium balances reasoning quality, cost, and latency.
	EffortMedium EffortLevel = "medium"
	// EffortHigh favors reasoning quality over cost and latency.
	EffortHigh EffortLevel = "high"
	// EffortXHigh requests extra-high effort, or the highest supported effort.
	EffortXHigh EffortLevel = "xhigh"
	// EffortMax selects the highest supported reasoning effort.
	EffortMax EffortLevel = "max"
)

func (EffortLevel) Validate ΒΆ added in v0.11.0

func (l EffortLevel) Validate() error

Validate ensures the effort level is valid.

type EmbeddingRequest ΒΆ added in v0.10.2

type EmbeddingRequest struct {
	// Inputs contains one text or document block per output vector.
	Inputs []Request `json:"inputs"`
	// Dimensions is the requested vector size, or zero for the model default.
	// Support depends on the provider and model.
	Dimensions int `json:"dimensions,omitzero"`
}

EmbeddingRequest embeds a batch of content with the client's configured model. Include any task prefixes required by that model in each request's Text.

func (*EmbeddingRequest) Validate ΒΆ added in v0.10.2

func (r *EmbeddingRequest) Validate() error

Validate implements Validatable.

type EmbeddingResponse ΒΆ added in v0.10.2

type EmbeddingResponse struct {
	Embeddings [][]float32 `json:"embeddings"`
	// Usage is zero when the provider doesn't report token counts.
	Usage Usage `json:"usage,omitzero"`
}

EmbeddingResponse contains one vector per input request, in input order. Values are preserved as returned by the provider. Normalization depends on the provider and model; callers comparing vectors must account for that contract.

func (*EmbeddingResponse) Validate ΒΆ added in v0.10.2

func (r *EmbeddingResponse) Validate() error

Validate implements Validatable. It checks uniform dimensions and finite, nonzero vectors without normalizing their values.

type FinishReason ΒΆ

type FinishReason string

FinishReason is the reason why the model stopped generating tokens.

It can be one of the well known below or a custom value.

const (
	// FinishedStop means the LLM was done for the turn. Some providers confuse it with
	// FinishedStopSequence.
	FinishedStop FinishReason = "stop"
	// FinishedLength means the model reached the maximum number of tokens allowed as set in
	// GenOptionText.MaxTokens or as limited by the provider.
	FinishedLength FinishReason = "length"
	// FinishedToolCalls means the model called one or multiple tools and needs the replies to continue the turn.
	FinishedToolCalls FinishReason = "tool_calls"
	// FinishedStopSequence means the model stopped because it saw a stop word as listed in GenOptionText.Stop.
	FinishedStopSequence FinishReason = "stop"
	// FinishedContentFilter means the model stopped because the reply got caught by a content filter.
	FinishedContentFilter FinishReason = "content_filter"
	// Pending means that it's not finished yet. For use with ProviderGenAsync.
	Pending FinishReason = "pending"
)

type GenOption ΒΆ added in v0.2.0

type GenOption interface {
	// Validate ensures the options object is valid.
	Validate() error
}

GenOption is options that can be provided to Provider.GenSync and Provider.GenStream.

type GenOptionAudio ΒΆ added in v0.2.0

type GenOptionAudio struct {
	// contains filtered or unexported fields
}

GenOptionAudio specifies audio generation options.

func (*GenOptionAudio) Validate ΒΆ added in v0.2.0

func (o *GenOptionAudio) Validate() error

Validate implements Validatable.

type GenOptionImage ΒΆ added in v0.2.0

type GenOptionImage struct {
	Width  int
	Height int
	// contains filtered or unexported fields
}

GenOptionImage is a list of frequent options supported by most ProviderDoc. Each provider is free to support more options through a specialized struct.

func (*GenOptionImage) Validate ΒΆ added in v0.2.0

func (o *GenOptionImage) Validate() error

Validate ensures the completion options are valid.

type GenOptionPollInterval ΒΆ

type GenOptionPollInterval time.Duration

GenOptionPollInterval is the time interval to poll generation progress when using GenSync.

func (GenOptionPollInterval) Validate ΒΆ

func (p GenOptionPollInterval) Validate() error

Validate ensures the poll interval is valid.

type GenOptionSeed ΒΆ added in v0.2.0

type GenOptionSeed int64

GenOptionSeed is the seed for the random number generator.

func (GenOptionSeed) Validate ΒΆ added in v0.2.0

func (s GenOptionSeed) Validate() error

Validate ensures the seed is valid.

type GenOptionText ΒΆ added in v0.2.0

type GenOptionText struct {
	// Temperature adjust the creativity of the sampling. Generally between 0 and 2.
	Temperature float64
	// TopP adjusts correctness sampling between 0 and 1. The higher the more diverse the output.
	TopP float64
	// MaxTokens is the maximum number of tokens to generate. Used to limit it
	// lower than the default maximum, for budget reasons.
	MaxTokens int64
	// TopLogprobs requests to return the top logprobs in the reply.
	TopLogprobs int64
	// SystemPrompt is the prompt to use for the system role.
	SystemPrompt string

	// TopK adjusts sampling where only the N first candidates are considered.
	TopK int64
	// Stop is the list of tokens to stop generation.
	Stop []string

	// ReplyAsJSON enforces the output to be valid JSON, any JSON. It is
	// important to tell the model to reply in JSON in the prompt itself.
	ReplyAsJSON bool
	// DecodeAs enforces a reply with a specific JSON structure. It must be either a JSON object or array
	// that can be decoded by encoding/json and can have jsonschema tags, usually a pointer to a struct, or
	// a JSONSchema.
	//
	// Not every provider accepts an array at the root of the reply, e.g. OpenAI and Anthropic require an
	// object. Wrap the slice in a struct with a single field when the provider rejects it.
	//
	// It is important to request the model to "reply in JSON" in the prompt itself.
	//
	// If you use a JSONSchema, it will be used to validate the reply.
	//
	// If you use a pointer to a struct, it is recommended to use jsonschema_description tags to describe each
	// field or argument. Use jsonschema:"enum=..." to enforce a specific value within a set. Use omitempty to
	// make the field optional. See https://github.com/invopop/jsonschema#example for more examples.
	DecodeAs any
	// contains filtered or unexported fields
}

GenOptionText is a list of frequent options supported by most Provider with text output modality. Each provider is free to support more options through a specialized struct.

The first group are options supported by (nearly) all providers.

The second group are options supported only by some providers. Using them may cause the chat operation to return a base.ErrNotsupported.

The third group are options supported by a few providers and a few models on each, that will slow down generation (increase latency) and will increase token use (cost).

func (*GenOptionText) DecodeSchema ΒΆ added in v0.6.1

func (o *GenOptionText) DecodeSchema() (JSONSchema, error)

DecodeSchema returns the JSONSchema for the DecodeAs field.

If DecodeAs is a JSONSchema instance, it is returned directly. Otherwise it uses invopop/jsonschema to construct the schema from the type's fields and tags.

func (*GenOptionText) Validate ΒΆ added in v0.2.0

func (o *GenOptionText) Validate() error

Validate ensures the completion options are valid.

type GenOptionTools ΒΆ added in v0.2.0

type GenOptionTools struct {
	// Tools is the list of tools that the LLM can request to call.
	Tools []ToolDef
	// Force tells the LLM a tool call must be done, or not.
	Force ToolCallRequest
}

GenOptionTools specifies tool calling options.

func (*GenOptionTools) Validate ΒΆ added in v0.2.0

func (o *GenOptionTools) Validate() error

Validate ensures the completion options are valid.

type GenOptionVideo ΒΆ added in v0.2.0

type GenOptionVideo struct {
	// Duration of the video to generate, if supported.
	//
	// Veo 2 supports only between 5 and 8 seconds and Veo 3 only supports 8 seconds.
	Duration time.Duration
	// contains filtered or unexported fields
}

GenOptionVideo specifies video generation options.

func (*GenOptionVideo) Validate ΒΆ added in v0.2.0

func (o *GenOptionVideo) Validate() error

Validate implements Validatable.

type GenOptionWeb ΒΆ added in v0.2.0

type GenOptionWeb struct {
	// Search specifies if websearch should be enabled. It is generally disabled by default.
	//
	// # Warning
	//
	// This will become a structure to provide information about included and excluded domains, and the user's
	// location.
	Search bool
	// Fetch specifies if web fetch should be enabled. When enabled, the LLM can fetch content from URLs.
	//
	// Currently supported by Anthropic and Gemini.
	Fetch bool
}

GenOptionWeb specifies web access options.

func (*GenOptionWeb) Validate ΒΆ added in v0.2.0

func (o *GenOptionWeb) Validate() error

Validate implements GenOption.

type JSONSchema ΒΆ added in v0.6.1

type JSONSchema = json.RawMessage

JSONSchema is a JSON Schema document encoded as raw JSON bytes.

type Job ΒΆ

type Job string

Job is a pending job.

type Logprob ΒΆ

type Logprob struct {
	ID      int64   `json:"id,omitzero"`   // Input token ID.
	Text    string  `json:"text,omitzero"` // Text in UTF-8.
	Logprob float64 `json:"logprob"`       // Log probability of the token. It should normally be non-zero but sometimes it is.
}

Logprob represents a single log probability information for a token.

One of ID or Text must be set.

func (*Logprob) GoString ΒΆ

func (l *Logprob) GoString() string

GoString returns a JSON representation of the reply for debugging purposes.

func (*Logprob) Validate ΒΆ added in v0.4.0

func (l *Logprob) Validate() error

Validate ensures the logprob is valid.

type Message ΒΆ

type Message struct {
	// Requests is the content from the user.
	//
	// It is normally a single message. It is more frequently multiple items when using multi-modal content.
	Requests []Request `json:"request,omitzero"`
	// User must only be used when sent by the user. Only some provider (e.g. OpenAI, Groq, DeepSeek) support it.
	User string `json:"user,omitzero"`

	// Replies is the message from the LLM.
	//
	// It is generally a single reply with text or a tool call. Some models can emit multiple content blocks,
	// either multi modal or multiple text blocks: a code block and a different block with an explanantion. Some
	// models can emit multiple tool calls at once.
	Replies []Reply `json:"reply,omitzero"`

	// ToolCallResult is the result for a tool call that the LLM requested to make.
	//
	// These messages are generated by the "computer".
	ToolCallResults []ToolCallResult `json:"tool_call_results,omitzero"`
	// contains filtered or unexported fields
}

Message is a part of an exchange with a LLM.

It is effectively a union, with the exception of the User field that can be set with In.

func NewTextMessage ΒΆ

func NewTextMessage(text string) Message

NewTextMessage is a shorthand function to create a Message with a single text block.

func (*Message) Accumulate ΒΆ

func (m *Message) Accumulate(mf *Reply) error

Accumulate adds a Reply to the message being streamed.

It is used by GenStream. There's generally no need to call it by end users.

func (*Message) Decode ΒΆ

func (m *Message) Decode(x any) error

Decode decodes the JSON message into the struct.

For generation replies, request JSON with ReplyAsJSON or DecodeAs in GenOptionText.

Note: this doesn't verify the type is the same as specified in GenOptionText.DecodeAs.

func (*Message) DoToolCalls ΒΆ

func (m *Message) DoToolCalls(ctx context.Context, tools []ToolDef) (Message, error)

DoToolCalls processes all the ToolCall in the Reply if any.

Returns a Message to be added back to the list of messages, only if msg.IsZero() is true.

func (*Message) GoString ΒΆ

func (m *Message) GoString() string

GoString returns a JSON representation of the reply for debugging purposes.

func (*Message) IsZero ΒΆ

func (m *Message) IsZero() bool

IsZero returns true if the message is empty.

func (*Message) Reasoning ΒΆ

func (m *Message) Reasoning() string

Reasoning returns all the reasoning concatenated, if any.

func (*Message) Role ΒΆ

func (m *Message) Role() string

Role returns one of "user", "assistant" or "computer".

func (*Message) String ΒΆ

func (m *Message) String() string

String is a short hand to get the request or reply content as text.

It ignores reasoning or multi-modal content.

func (*Message) UnmarshalJSON ΒΆ

func (m *Message) UnmarshalJSON(b []byte) error

UnmarshalJSON adds validation during decoding.

func (*Message) Validate ΒΆ

func (m *Message) Validate() error

Validate ensures the message is valid.

type Messages ΒΆ

type Messages []Message

Messages is a chronological list of messages in an exchange with a LLM.

Adjacent messages may have the same role. Tool calls may remain unresolved, and user messages and other tool results may intervene before their results arrive. Providers may impose additional constraints. Conversion must preserve event order without merging messages or inventing tool results.

func (Messages) Validate ΒΆ

func (m Messages) Validate() error

Validate ensures the messages are valid.

type Modalities ΒΆ

type Modalities []scoreboard.Modality

Modalities represents the modality supported by the provider in a specific scenario. It can be multiple modalities in multi-modals scenarios.

func (Modalities) String ΒΆ

func (m Modalities) String() string

func (Modalities) Validate ΒΆ

func (m Modalities) Validate() error

Validate implements Validatable.

type Modality ΒΆ

type Modality = scoreboard.Modality

Modality is a modality supported by the provider.

It is aliased from scoreboard.Modality to avoid a circular dependency.

type Model ΒΆ

type Model interface {
	GetID() string
	String() string
	// Context returns the number of tokens the model can process as input.
	Context() int64
}

Model represents a served model by the provider.

Use Provider.ListModels() to get a list of models.

type NoulCriteria ΒΆ added in v0.10.0

type NoulCriteria struct {
	// True describes what a yes (value near 1) means.
	True DecisionContent `json:"true,omitzero"`
	// False describes what a no (value near 0) means.
	False DecisionContent `json:"false,omitzero"`
}

NoulCriteria describes what the yes and the no answers of a noul question mean.

func (*NoulCriteria) Validate ΒΆ added in v0.10.0

func (c *NoulCriteria) Validate() error

Validate ensures the criteria are valid.

type Object ΒΆ added in v0.10.0

type Object map[string]any

Object is DecisionContent that is a JSON object.

The values are any, like Reply.Opaque: they must be JSON encodable. Go has no recursive JSON value type that does not require wrapping every scalar, so a nested map[string]any, []any or a struct with JSON tags is passed as is. It is the equivalent of the SDKs' Mapping[str, JSONValue | None].

func (Object) Validate ΒΆ added in v0.10.0

func (o Object) Validate() error

Validate implements internal.Validatable.

Every value is validated so that the error names the offending key.

type Provider ΒΆ

type Provider interface {
	// Close releases resources held by the provider.
	io.Closer

	// Name returns the name of the provider.
	Name() string
	// ModelID returns the model currently used by the provider. It can be an empty string.
	ModelID() string
	// OutputModalities returns the output modalities supported by this specific client configuration.
	//
	// This states what kind of output the model will generate (text, audio, image, video, decision). It varies per
	// provider and models. The vast majority of providers and models support only output modality like
	// text-only, image-only, etc.
	OutputModalities() Modalities
	// Capabilities returns the optional capabilities this provider supports.
	Capabilities() ProviderCapabilities
	// Scoreboard returns what the provider supports.
	//
	// Some models have more features than others, e.g. some models may be text-only while others have vision or
	// audio support.
	//
	// The client code may be the limiting factor for some models, and not the provider itself.
	//
	// Capability results come from smoke tests; voice catalogs describe
	// provider-advertised choices.
	Scoreboard() scoreboard.Score
	// HTTPClient returns the underlying http client. It may be necessary to use it to fetch the results from
	// the provider. An example is retrieving Veo 3 generated videos from Gemini requires the authentication
	// headers to be set.
	HTTPClient() *http.Client

	// GenSync runs generation synchronously.
	//
	// Multiple options can be mixed together, both standard ones like *GenOptionImage, *GenOptionText,
	// *GenOptionTools and provider-specialized options struct, e.g. *anthropic.GenOptionText, *gemini.GenOption.
	GenSync(ctx context.Context, msgs Messages, opts ...GenOption) (Result, error)
	// GenStream runs generation synchronously, yielding the fragments of replies as the server sends them.
	//
	// No need to accumulate the fragments into a Message since the Result contains the accumulated message.
	GenStream(ctx context.Context, msgs Messages, opts ...GenOption) (iter.Seq[Reply], func() (Result, error))
	// SystemOne answers typed questions about a state without text generation or streaming.
	// It uses the client's configured model. The request is not modified.
	// Requires ProviderCapabilities.SystemOne. Returns base.ErrNotSupported otherwise.
	SystemOne(ctx context.Context, req *SystemOneRequest) (*SystemOneResponse, error)
	// Embed returns one vector per input request in order using the configured model.
	// Configure an explicit embedding model; automatic generation model selection
	// does not choose one. Content blocks and options are not modified; document
	// readers may be read and sought. Input size limits depend
	// on the provider and model. Returns base.ErrNotSupported when unsupported.
	// Provider-specific raw APIs expose additional embedding controls.
	// Capabilities().Embed describes endpoint implementation; the scoreboard
	// Embedding lookup describes tested support for the configured model.
	Embed(ctx context.Context, in *EmbeddingRequest) (*EmbeddingResponse, error)
	// ConnectRealtime opens a bidirectional conversation and waits for setup acceptance.
	// Requires Capabilities().Realtime. Unsupported session features return base.ErrNotSupported.
	ConnectRealtime(ctx context.Context, opts ...RealtimeOption) (RealtimeSession, error)
	// ListModels returns the list of models the provider supports. Not all providers support it, some will
	// return an ErrorNotSupported. For local providers like llamacpp and ollama, they may return only the
	// model currently loaded.
	ListModels(ctx context.Context) ([]Model, error)

	// GenAsync requests a generation and returns a pending job that can be polled.
	//
	// Requires ProviderCapabilities.GenAsync to be set. Returns base.ErrNotSupported otherwise.
	GenAsync(ctx context.Context, msgs Messages, opts ...GenOption) (Job, error)
	// PokeResult requests the state of the job.
	//
	// When the job is still pending, Result.Usage.FinishReason is Pending.
	//
	// Requires ProviderCapabilities.GenAsync to be set. Returns base.ErrNotSupported otherwise.
	PokeResult(ctx context.Context, job Job) (Result, error)
	// CacheAddRequest caches a request.
	//
	// Requires ProviderCapabilities.Caching to be set. Returns base.ErrNotSupported otherwise.
	//
	// # Warning
	//
	// May be changed in the future.
	CacheAddRequest(ctx context.Context, msgs Messages, name, displayName string, ttl time.Duration, opts ...GenOption) (string, error)
	// CacheList lists the caches entries.
	//
	// Requires ProviderCapabilities.Caching to be set. Returns base.ErrNotSupported otherwise.
	//
	// # Warning
	//
	// May be changed in the future.
	CacheList(ctx context.Context) ([]CacheEntry, error)
	// CacheDelete deletes a cache entry.
	//
	// Requires ProviderCapabilities.Caching to be set. Returns base.ErrNotSupported otherwise.
	//
	// # Warning
	//
	// May be changed in the future.
	CacheDelete(ctx context.Context, name string) error
}

Provider is the base interface that all provider interfaces embed.

The first group contains local methods. Calling these methods will not make an HTTP request.

The second group is supported by the majority of providers.

The rest is supported by a limited number of providers.

Example (Embed) ΒΆ
package main

import (
	"context"
	"fmt"
	"log"

	"github.com/maruel/genai"
	"github.com/maruel/genai/providers/gemini"
)

func main() {
	ctx := context.Background()
	c, err := gemini.New(ctx, genai.ProviderOptionModel("gemini-embedding-2"))
	if c != nil {
		defer func() {
			if err := c.Close(); err != nil {
				log.Printf("Close: %v", err)
			}
		}()
	}
	if err != nil {
		log.Fatal(err)
	}
	if !c.Capabilities().Embed {
		log.Fatal("embedding endpoint is unavailable")
	}
	sb := c.Scoreboard()
	if f := sb.Embedding(c.ModelID()); f != nil {
		fmt.Printf("Qualified default size: %d\n", f.Dimensions)
	} else {
		fmt.Println("Embedding model support is unmeasured")
	}
	out, err := c.Embed(ctx, &genai.EmbeddingRequest{
		Inputs:     []genai.Request{{Text: "A kitten plays with yarn."}, {Text: "A cat plays with string."}},
		Dimensions: 768,
	})
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("Returned %d vectors with %d dimensions\n", len(out.Embeddings), len(out.Embeddings[0]))
}
Example (GenSync_audio) ΒΆ
package main

import (
	"context"
	"fmt"
	"log"
	"os"
	"strings"

	"github.com/maruel/genai"
	"github.com/maruel/genai/providers/gemini"
)

func main() {
	// Supported by Gemini, OpenAI.

	// Using a free small model for testing.
	// See https://ai.google.dev/gemini-api/docs/models/gemini?hl=en
	ctx := context.Background()
	c, err := gemini.New(ctx, genai.ProviderOptionModel("gemini-2.5-flash-lite"))
	if c != nil {
		defer func() {
			if err := c.Close(); err != nil {
				log.Printf("Close: %v", err)
			}
		}()
	}
	if err != nil {
		log.Print(err)
		return
	}
	f, err := os.Open("internal/internaltest/testdata/mystery_word.mp3")
	if err != nil {
		log.Print(err)
		return
	}
	defer func() { _ = f.Close() }()
	msgs := genai.Messages{
		{
			Requests: []genai.Request{
				{Text: "What is the word said? Reply with only the word."},
				{Doc: genai.Doc{Src: f}},
			},
		},
	}
	resp, err := c.GenSync(ctx, msgs)
	if err != nil {
		log.Print(err)
		return
	}
	fmt.Printf("Heard: %v\n", strings.TrimRight(strings.ToLower(resp.String()), "."))
	// This would Output: Heard: orange
}
Example (GenSync_pdf) ΒΆ
package main

import (
	"context"
	"fmt"
	"log"
	"os"
	"strings"

	"github.com/maruel/genai"
	"github.com/maruel/genai/providers/gemini"
)

func main() {
	// Supported by Anthropic, Gemini, Mistral, OpenAI.

	// Using a free small model for testing.
	// See https://ai.google.dev/gemini-api/docs/models/gemini?hl=en
	ctx := context.Background()
	c, err := gemini.New(ctx, genai.ProviderOptionModel("gemini-2.5-flash-lite"))
	if c != nil {
		defer func() {
			if err := c.Close(); err != nil {
				log.Printf("Close: %v", err)
			}
		}()
	}
	if err != nil {
		log.Print(err)
		return
	}
	f, err := os.Open("internal/internaltest/testdata/hidden_word.pdf")
	if err != nil {
		log.Print(err)
		return
	}
	defer func() { _ = f.Close() }()
	msgs := genai.Messages{
		{
			Requests: []genai.Request{
				{Text: "What is the word? Reply with only the word."},
				{Doc: genai.Doc{Src: f}},
			},
		},
	}
	resp, err := c.GenSync(ctx, msgs)
	if err != nil {
		log.Print(err)
		return
	}
	fmt.Printf("Hidden word in PDF: %v\n", strings.ToLower(resp.String()))
	// This would Output: Hidden word in PDF: orange
}
Example (GenSync_video) ΒΆ
package main

import (
	"context"
	"fmt"
	"log"
	"os"
	"strings"

	"github.com/maruel/genai"
	"github.com/maruel/genai/providers/gemini"
)

func main() {
	// Supported by Gemini, TogetherAI.

	// Using a free small model for testing.
	// See https://ai.google.dev/gemini-api/docs/models/gemini?hl=en
	ctx := context.Background()
	c, err := gemini.New(ctx, genai.ProviderOptionModel("gemini-2.5-flash"))
	if c != nil {
		defer func() {
			if err := c.Close(); err != nil {
				log.Printf("Close: %v", err)
			}
		}()
	}
	if err != nil {
		log.Print(err)
		return
	}
	f, err := os.Open("internal/internaltest/testdata/animation.mp4")
	if err != nil {
		log.Print(err)
		return
	}
	defer func() { _ = f.Close() }()
	// TogetherAI seems to require separate messages for text and images.
	msgs := genai.Messages{
		genai.NewTextMessage("What is the word? Reply with exactly and only one word."),
		{Requests: []genai.Request{{Doc: genai.Doc{Src: f}}}},
	}
	resp, err := c.GenSync(ctx, msgs)
	if err != nil {
		log.Print(err)
		return
	}
	fmt.Printf("Saw: %v\n", strings.ToLower(resp.String()))
	// This would Output: Saw: banana
}
Example (GenSync_vision) ΒΆ
package main

import (
	"bytes"
	"context"
	"fmt"
	"log"
	"os"

	"github.com/maruel/genai"
	"github.com/maruel/genai/providers/gemini"
)

func main() {
	// Supported by Anthropic, Gemini, Groq, Mistral, Ollama, OpenAI, TogetherAI.

	// Using a free small model for testing.
	// See https://ai.google.dev/gemini-api/docs/models/gemini?hl=en
	ctx := context.Background()
	c, err := gemini.New(ctx, genai.ProviderOptionModel("gemini-2.5-flash-lite"))
	if c != nil {
		defer func() {
			if err := c.Close(); err != nil {
				log.Printf("Close: %v", err)
			}
		}()
	}
	if err != nil {
		log.Print(err)
		return
	}
	bananaJpg, err := os.ReadFile("internal/internaltest/testdata/banana.jpg")
	if err != nil {
		log.Print(err)
		return
	}
	msgs := genai.Messages{
		{
			Requests: []genai.Request{
				{Text: "Is it a banana? Reply with only the word yes or no."},
				{Doc: genai.Doc{Filename: "banana.jpg", Src: bytes.NewReader(bananaJpg)}},
			},
		},
	}
	resp, err := c.GenSync(ctx, msgs)
	if err != nil {
		log.Print(err)
		return
	}
	fmt.Printf("Banana: %v\n", resp.String())
	// This would Output: Banana: yes
}

type ProviderCapabilities ΒΆ

type ProviderCapabilities struct {
	// Realtime indicates that the client implements bidirectional realtime sessions,
	// independent of its configured model. Session capabilities are negotiated by
	// ConnectRealtime and reported through RealtimeSession.Info.
	Realtime bool
	// Embed indicates that the client implements the embedding endpoint, independent
	// of its configured model. scoreboard.Score.Embedding reports qualified support;
	// a missing measurement means unknown, not unsupported.
	Embed bool
	// SystemOne indicates the provider implements typed decision inference.
	SystemOne bool
	// GenAsync indicates the provider supports GenAsync and PokeResult for batch operations.
	GenAsync bool
	// Caching indicates the provider supports CacheAddRequest, CacheList, and CacheDelete.
	Caching bool
	// contains filtered or unexported fields
}

ProviderCapabilities describes optional capabilities implemented by a client.

type ProviderOption ΒΆ

type ProviderOption interface {
	Validate() error
}

ProviderOption is an option for provider constructors.

type ProviderOptionAPIKey ΒΆ

type ProviderOptionAPIKey string

ProviderOptionAPIKey provides an API key to authenticate to the server.

Most providers require an API key, and the client will look at an environment variable "<PROVIDER>_API_KEY" to use as a default value if unspecified.

func (ProviderOptionAPIKey) Validate ΒΆ

func (p ProviderOptionAPIKey) Validate() error

Validate implements Validatable.

type ProviderOptionModalities ΒΆ

type ProviderOptionModalities Modalities

ProviderOptionModalities is the list of output modalities you request the model to support.

Most provider support text output only. Most models support output of only one modality, either text, image, audio or video. But a few models do support both text and images.

When unspecified, the provider defaults to all modalities supported by the provider and the selected model.

Even when Model is set to a specific model ID, a ListModels call may be made to discover its supported output modalities for providers that support multiple output modalities.

ProviderOptionModalities can be set without a Model to test if a provider supports a modality without causing a ListModels call.

func (ProviderOptionModalities) Validate ΒΆ

func (p ProviderOptionModalities) Validate() error

Validate implements Validatable.

type ProviderOptionModel ΒΆ

type ProviderOptionModel string

ProviderOptionModel specifies which model to use.

For automatic model selection, use the predefined constants ModelCheap, ModelGood, or ModelSOTA directly as provider options. The provider internally calls ListModels() to discover models and select based on its heuristics. Providers that do not support ListModels (e.g. bfl) use a hardcoded list.

When unspecified, no model is selected. This is useful when only calling ListModels(). Generation calls will fail.

Keep in mind that as providers cycle through new models, it's possible a specific model ID is not available anymore or that the default model changes.

const (
	// ModelCheap requests the provider to automatically select the cheapest model it can find.
	ModelCheap ProviderOptionModel = "CHEAP"
	// ModelGood requests the provider to automatically select a good every day model that has a good
	// performance/cost trade-off.
	ModelGood ProviderOptionModel = "GOOD"
	// ModelSOTA requests the provider to automatically select the best state-of-the-art model
	// it can find.
	ModelSOTA ProviderOptionModel = "SOTA"
)

Model markers for automatic model selection. These are ProviderOptionModel values that can be passed directly to provider constructors.

func (ProviderOptionModel) Validate ΒΆ

func (p ProviderOptionModel) Validate() error

Validate implements Validatable.

type ProviderOptionPreloadedModels ΒΆ

type ProviderOptionPreloadedModels []Model

ProviderOptionPreloadedModels is a list of models that are preloaded into the provider, to replace the call to ListModels, for example with automatic model selection and modality detection.

This is mostly used for unit tests or repeated client creation to save on HTTP requests.

func (ProviderOptionPreloadedModels) Validate ΒΆ

func (p ProviderOptionPreloadedModels) Validate() error

Validate implements Validatable.

type ProviderOptionRemote ΒΆ

type ProviderOptionRemote string

ProviderOptionRemote is the remote address to access the service.

It is mostly used by locally hosted services (llamacpp, ollama) or for generic client (openaicompatible).

func (ProviderOptionRemote) Validate ΒΆ

func (p ProviderOptionRemote) Validate() error

Validate implements Validatable.

type ProviderOptionStarterWrapper ΒΆ added in v0.4.1

type ProviderOptionStarterWrapper func(Starter) Starter

ProviderOptionStarterWrapper wraps the subprocess starter used by CLI-based providers.

This is the subprocess analog of ProviderOptionTransportWrapper: it lets callers intercept process creation for recording, replay, or testing.

func (ProviderOptionStarterWrapper) Validate ΒΆ added in v0.4.1

func (p ProviderOptionStarterWrapper) Validate() error

Validate implements Validatable.

type ProviderOptionTransportWrapper ΒΆ

type ProviderOptionTransportWrapper func(http.RoundTripper) http.RoundTripper

ProviderOptionTransportWrapper wraps the HTTP transport used by the provider.

This is useful for adding middleware like logging, tracing, or HTTP recording for tests.

func (ProviderOptionTransportWrapper) Validate ΒΆ

Validate implements Validatable.

type ProviderPing ΒΆ

type ProviderPing interface {
	Provider
	// Ping enables confirming that the provider is accessible, without incurring cost. This is useful for local
	// providers to detect if they are accessible or not.
	Ping(ctx context.Context) error
}

ProviderPing represents a provider that you can ping.

type ProviderScoreboardVariants ΒΆ added in v0.3.0

type ProviderScoreboardVariants interface {
	ScoreboardVariants() []ScoreboardVariant
}

ProviderScoreboardVariants is optionally implemented by providers that have multiple regional backends with different model catalogs.

When implemented, documentation tools generate a separate table per variant instead of a single table.

type ProviderUnwrap ΒΆ

type ProviderUnwrap interface {
	Unwrap() Provider
}

ProviderUnwrap is exposed when the Provider is actually a wrapper around another one, like adapters.ProviderReasoning or ProviderUsage. This is useful when looking for other interfaces.

type Question ΒΆ added in v0.10.0

type Question struct {
	// Type is how the state is evaluated. It selects which of Noul, Choice or Score must be set.
	Type QuestionType
	// Instructions is what the model should decide or rate. Providers may require it.
	Instructions DecisionContent
	// Noul describes what the yes and the no outcomes mean. It can only be set for QuestionNoul, where it
	// is optional.
	//
	// Recommended reading: https://docs.typesafe.ai/primitives/noul
	Noul *NoulCriteria
	// Choice maps the options of a choice question to their descriptions. It can only be set for
	// QuestionChoice, where at least one option is required. Use nil for an option that needs no extra
	// detail.
	//
	// Recommended reading: https://docs.typesafe.ai/primitives/choice
	Choice map[string]DecisionContent
	// Score lists the levels of a score question rubric, in order, starting at level 0. It can only be set
	// for QuestionScore, where at least one level is required. Use nil for an undescribed level.
	// Providers may impose stricter limits.
	//
	// Recommended reading: https://docs.typesafe.ai/primitives/score
	Score []DecisionContent
}

Question is a typed question about a state.

Exactly one of Noul, Choice or Score can be set, the one matching Type. Instructions is shared by the three types.

Type is explicit instead of inferred from the field that is set: a choice or score question whose criteria are nil at runtime, which append and conditional map building produce, would otherwise be silently asked as a noul question instead of being rejected.

Recommended reading: https://docs.typesafe.ai/primitives/advanced

func (*Question) MarshalJSON ΒΆ added in v0.10.0

func (q *Question) MarshalJSON() ([]byte, error)

MarshalJSON implements json.Marshaler.

func (*Question) Validate ΒΆ added in v0.10.0

func (q *Question) Validate() error

Validate checks shared content and union consistency, not provider-specific limits.

type QuestionType ΒΆ added in v0.10.0

type QuestionType string

QuestionType is the type of a Question.

const (
	// QuestionNoul is a yes/no question. The answer is the probability that the answer is yes.
	QuestionNoul QuestionType = "noul"
	// QuestionChoice picks one option from a set defined by the question.
	QuestionChoice QuestionType = "choice"
	// QuestionScore rates the state along an ordered rubric.
	QuestionScore QuestionType = "score"
)

Question types.

type Questions ΒΆ added in v0.10.0

type Questions map[string]*Question

Questions is a set of questions to ask about one state, keyed by the name to report each answer under.

The names are chosen by the caller. They are not sent to the model, they are only used to key the answers.

Set SystemOneRequest.Questions directly. SystemOneResponse.Answers holds the typed answers under the same dynamic names. Entries must not be nil.

Example ΒΆ

ExampleQuestions declares typed questions under caller-selected names.

package main

import (
	"encoding/json"
	"fmt"
	"log"

	"github.com/maruel/genai"
)

func main() {
	questions := genai.Questions{
		"billing": {Type: genai.QuestionNoul, Instructions: genai.Text("Is this request about billing?")},
		"tone":    {Type: genai.QuestionChoice, Instructions: genai.Text("What is the tone of the customer?"), Choice: map[string]genai.DecisionContent{"calm": nil, "angry": nil}},
	}

	raw, err := json.MarshalIndent(questions, "", "  ")
	if err != nil {
		log.Print(err)
		return
	}
	fmt.Printf("%s\n", raw)
}
Output:
{
  "billing": {
    "type": "noul",
    "instructions": "Is this request about billing?"
  },
  "tone": {
    "type": "choice",
    "instructions": "What is the tone of the customer?",
    "criteria": {
      "angry": null,
      "calm": null
    }
  }
}

func (Questions) Validate ΒΆ added in v0.10.0

func (q Questions) Validate() error

Validate ensures the questions are valid.

type RateLimit ΒΆ

type RateLimit struct {
	Type      RateLimitType
	Period    RateLimitPeriod
	Limit     int64
	Remaining int64
	Reset     time.Time
}

RateLimit contains the limit, remaining, and reset values for a metric.

func (*RateLimit) String ΒΆ

func (r *RateLimit) String() string

func (*RateLimit) Validate ΒΆ

func (r *RateLimit) Validate() error

Validate ensures the rate limit is valid.

type RateLimitPeriod ΒΆ

type RateLimitPeriod int32

RateLimitPeriod defines the time period for a rate limit.

const (
	PerOther RateLimitPeriod = iota // For non-standard periods
	PerMinute
	PerDay
	PerMonth
)

RateLimitPeriod values.

type RateLimitType ΒΆ

type RateLimitType int32

RateLimitType defines the type of rate limit.

const (
	Requests RateLimitType = iota + 1
	Tokens
)

RateLimitType values.

type RealtimeCancel ΒΆ added in v0.11.0

type RealtimeCancel struct {
	ResponseID string
}

RealtimeCancel stops generation, optionally targeting ResponseID. It does not remove conversation history or queued playback.

func (RealtimeCancel) Validate ΒΆ added in v0.11.0

func (RealtimeCancel) Validate() error

Validate implements RealtimeInput.

type RealtimeCapabilities ΒΆ added in v0.11.0

type RealtimeCapabilities struct {
	AudioCommit    bool
	CancelResponse bool
	ToolResults    bool
	Truncate       bool
	Update         RealtimeUpdateCapabilities
}

RealtimeCapabilities describes the operations this configured session accepts. A false field means the provider rejects that command with base.ErrNotSupported. Configured features are reported by RealtimeSessionInfo, not here.

AudioCommit reports that RealtimeInputContent.Commit commits manual-turn audio. ToolResults reports that the session accepts tool result batches.

type RealtimeContent ΒΆ added in v0.11.0

type RealtimeContent struct {
	// Audio is owned PCM at Info().OutputFormat for Audio events.
	Audio []byte
	// Text is a delta for Text and transcript events. When Final is true, Text
	// is the complete text and replaces accumulated deltas.
	Text string
	// Reasoning is a delta for Reasoning events.
	Reasoning string
	// Final marks Text as complete for Text and transcript events.
	Final bool
	// ToolCalls contains complete calls with validated JSON arguments and unique
	// nonempty IDs for ToolCalls events. Providers that omit call IDs must assign
	// session-unique IDs and preserve the original provider correlation internally.
	ToolCalls []ToolCall
}

RealtimeContent is the payload of a RealtimeEvent. The zero value means the event carries no content. At most one of Audio, Text, or ToolCalls is set. Final is valid only with Text.

func (*RealtimeContent) Validate ΒΆ added in v0.11.0

func (c *RealtimeContent) Validate() error

Validate rejects multiple payload fields or Final without Text.

type RealtimeError ΒΆ added in v0.11.0

type RealtimeError struct {
	// Code is the provider error code when reported.
	Code string
	// Message is a short, caller-safe description of the failure.
	Message string
	// EventID identifies the rejected client event when reported.
	EventID string
}

RealtimeError is an error reported by the provider. RealtimeEventError carries a recoverable one; setup and transport failures are returned directly.

func (*RealtimeError) Error ΒΆ added in v0.11.0

func (e *RealtimeError) Error() string

Error implements error.

type RealtimeEvent ΒΆ added in v0.11.0

type RealtimeEvent struct {
	// Kind identifies the event. GenerationDone means output generation ended,
	// not playback ended. TurnDone means the provider completed its turn, which
	// need not mean session idleness. Interrupted signals the caller to stop
	// playback.
	Kind RealtimeEventKind
	// ResponseID identifies the response within this session. It may be a provider
	// identity or an adapter-assigned identity based on provider turn boundaries.
	ResponseID string
	// ItemID identifies the conversation item when the provider supplies it.
	ItemID string
	// ContentIndex identifies the content part within ItemID.
	ContentIndex int
	// Content is the event payload. Its zero value means the event carries no
	// content.
	Content RealtimeContent
	// Usage is a non-cumulative snapshot for one response. Each Usage event
	// describes its own response; sum them to compute session usage. ResponseID
	// identifies the response when its ownership is known. Limits, when present,
	// replaces the previous rate-limit snapshot.
	Usage *Usage
	// Error carries a recoverable provider error for Error events. The session
	// remains usable. Terminal failures are returned by Receive instead.
	Error *RealtimeError
	// ProviderEvent is provider-specific information for this event. It is nil
	// when the provider supplies none. For RealtimeEventProvider, it is the
	// complete payload.
	ProviderEvent RealtimeProviderEvent
}

RealtimeEvent is one normalized event. Kind identifies the event; Content carries its payload. Provider-specific events use RealtimeEventProvider. Empty IDs mean unspecified. Adapters may assign session-local response identities when documented provider boundaries make correlation unambiguous.

type RealtimeEventKind ΒΆ added in v0.11.0

type RealtimeEventKind string

RealtimeEventKind identifies an event emitted by a session.

const (
	// RealtimeEventContentAudio carries one chunk of generated PCM audio at
	// Info().OutputFormat. Chunks may split words and utterances arbitrarily.
	RealtimeEventContentAudio RealtimeEventKind = "audio"
	// RealtimeEventContentText carries generated text. Content.Text is a delta unless
	// Final is true, in which case it replaces the accumulated deltas.
	RealtimeEventContentText RealtimeEventKind = "text"
	// RealtimeEventContentReasoning carries reasoning text that the provider
	// separates from the answer. Content.Reasoning is a delta.
	RealtimeEventContentReasoning RealtimeEventKind = "reasoning"
	// RealtimeEventContentToolCalls carries complete function calls from one response.
	// Arguments are validated JSON, and IDs are unique and nonempty. The caller
	// executes the calls and sends the results in RealtimeInputContent.ToolResults. The
	// turn cannot complete until it receives those results.
	RealtimeEventContentToolCalls RealtimeEventKind = "tool_calls"
	// RealtimeEventError reports a recoverable provider error. The session
	// remains usable.
	RealtimeEventError RealtimeEventKind = "error"
	// RealtimeEventGenerationDone reports that the provider ended generation for
	// one response. Playback may continue, and a response that only calls tools
	// also emits it. It does not end the turn. An interrupted response also emits
	// it before TurnDone.
	RealtimeEventGenerationDone RealtimeEventKind = "generation_done"
	// RealtimeEventInputSpeechEnd reports that provider voice activity detection
	// stopped detecting user speech. It occurs only with server turn detection;
	// manual turns disable provider VAD. ItemID may identify the user item. The
	// provider may separately commit audio or start a response when configured;
	// this event guarantees neither.
	RealtimeEventInputSpeechEnd RealtimeEventKind = "input_speech_end"
	// RealtimeEventInputSpeechStart reports that provider voice activity detection
	// detected user speech. It occurs only with server turn detection; manual
	// turns disable provider VAD. ItemID may identify the user item that speech
	// will create. It does not guarantee that a turn produces a response.
	RealtimeEventInputSpeechStart RealtimeEventKind = "input_speech_start"
	// RealtimeEventInterrupted reports that the provider stopped the active
	// response before normal completion. Stop and discard queued playback. The
	// session remains usable.
	RealtimeEventInterrupted RealtimeEventKind = "interrupted"
	// RealtimeEventProvider carries an event with no portable kind. ProviderEvent
	// holds the concrete provider type. Callers must ignore provider event types
	// they do not handle.
	RealtimeEventProvider RealtimeEventKind = "provider"
	// RealtimeEventTranscriptInput carries a transcription of user audio.
	// Content.Text is a delta unless Final is true, in which case it replaces the
	// accumulated deltas. Transcription is independent of model output and may
	// arrive out of order with respect to response events.
	RealtimeEventTranscriptInput RealtimeEventKind = "input_transcript"
	// RealtimeEventTranscriptOutput carries a transcription of generated audio.
	// Content.Text is a delta unless Final is true, in which case it replaces the
	// accumulated deltas. It is not generated text output; RealtimeEventContentText
	// carries that.
	RealtimeEventTranscriptOutput RealtimeEventKind = "output_transcript"
	// RealtimeEventTurnDone reports that the provider completed the current turn.
	// No more output follows for the current input unless the caller submits tool
	// results. It also follows an interrupted, cancelled, or failed response. The
	// session remains open.
	RealtimeEventTurnDone RealtimeEventKind = "turn_done"
	// RealtimeEventUsage reports token usage for one response and may include a
	// rate-limit snapshot. It is not cumulative; sum the token fields across
	// events for session totals. When Usage.Limits is set, it replaces the
	// previous limits.
	RealtimeEventUsage RealtimeEventKind = "usage"
)

RealtimeEventKind values. Each kind defines the RealtimeEvent fields that apply to it.

type RealtimeInput ΒΆ added in v0.11.0

type RealtimeInput interface {
	Validate() error
	// contains filtered or unexported methods
}

RealtimeInput is one command sent to RealtimeSession. Only the types in this package implement it.

type RealtimeInputContent ΒΆ added in v0.11.0

type RealtimeInputContent struct {
	// Audio appends PCM at Info().InputFormat. It may be empty only with Commit.
	Audio []byte
	// Doc adds a user turn carrying the document. Text may follow it.
	Doc Doc
	// Text adds a user text turn. It may be empty only with Generate.
	Text string
	// ToolResults answers completed calls. It always requests a response.
	ToolResults []ToolCallResult
	// Commit commits buffered audio. Alone it does not request a reply; with
	// Generate it commits before requesting one.
	Commit bool
	// Generate requests a response. It commits nonempty audio first.
	Generate bool
}

RealtimeInputContent sends one content command. Audio is selected by Audio or Commit; otherwise one of Doc or ToolResults is set, and Text may accompany Doc as one turn. The zero value is invalid.

Audio appends PCM at Info().InputFormat to the buffered user audio. Commit commits the buffer and may close the provider audio stream; an empty Commit commits audio sent earlier. Generate requests a reply and commits nonempty audio first when needed. Commit and Generate may be combined: Commit finalizes buffered audio, then Generate requests the response. With server turn detection, the provider may ignore Generate and reply on its own.

Doc adds a user turn carrying the document. Text, when set, follows it in the same turn and Generate requests the response. Each provider bounds the document size. A model that turns the document down reports a RealtimeEventError for the command.

Text adds a user text turn, or follows the document when Doc is set. Generate=false adds passive context; Generate=true submits the turn and requests a response. An empty Text with Generate=true requests a response from the current context. Providers without an explicit generation command may ignore it.

ToolResults answers completed calls. Send complete batches to preserve parallel call order. Tool results always request a response; Generate is ignored for this payload.

func (RealtimeInputContent) Validate ΒΆ added in v0.11.0

func (i RealtimeInputContent) Validate() error

Validate checks that one payload is set and that its flags are valid.

type RealtimeOption ΒΆ added in v0.11.0

type RealtimeOption interface {
	Validate() error
}

RealtimeOption configures a realtime session.

Portable options are RealtimeOptionInstructions, RealtimeOptionAudio, RealtimeOptionTools, RealtimeTurnDetection, and Modalities. Modalities selects output modalities and accepts only ModalityAudio and ModalityText; empty uses the provider's default output mode. Input is per command: audio, a document, text, or tool results.

A provider may define its own option type for settings that have no portable meaning. A provider rejects options it cannot honor rather than ignoring them.

type RealtimeOptionAudio ΒΆ added in v0.11.0

type RealtimeOptionAudio struct {
	Voice               string
	InputTranscription  bool
	OutputTranscription bool
}

RealtimeOptionAudio requests voice selection and audio transcripts. Transcription is opt-in and independent of generated text output. A false transcription field requests no transcription but does not promise that the provider sends no transcript events; callers must handle them either way.

func (*RealtimeOptionAudio) Validate ΒΆ added in v0.11.0

func (*RealtimeOptionAudio) Validate() error

Validate implements RealtimeOption.

type RealtimeOptionInstructions ΒΆ added in v0.11.0

type RealtimeOptionInstructions string

RealtimeOptionInstructions supplies session instructions.

func (RealtimeOptionInstructions) Validate ΒΆ added in v0.11.0

func (RealtimeOptionInstructions) Validate() error

Validate implements RealtimeOption.

type RealtimeOptionTools ΒΆ added in v0.11.0

type RealtimeOptionTools struct {
	Tools []ToolDef
}

RealtimeOptionTools declares session tools. The application owns their execution.

func (*RealtimeOptionTools) Validate ΒΆ added in v0.11.0

func (o *RealtimeOptionTools) Validate() error

Validate implements RealtimeOption.

type RealtimeProviderEvent ΒΆ added in v0.11.0

type RealtimeProviderEvent interface {
	// RealtimeEventName returns the provider's wire event name.
	RealtimeEventName() string
}

RealtimeProviderEvent is a provider-specific event payload. Providers define the concrete types and document the event names they represent. Providers use it when no portable event kind applies; callers ignore event kinds they do not handle.

type RealtimeSession ΒΆ added in v0.11.0

type RealtimeSession interface {
	io.Closer
	// Info returns immutable native formats, accepted input kinds, configured
	// features, and accepted operations after setup.
	Info() RealtimeSessionInfo
	// Send applies one complete command to the session and writes it to the
	// provider. It rejects invalid, unsupported, and out-of-order commands before
	// the first wire write. It serializes concurrent calls, translates the command
	// into the provider's wire messages, and updates local correlation only after
	// every frame is written. Input buffers are borrowed until Send returns and are
	// not retained. A successful return means written, not accepted by the
	// provider. A write failure may leave a partial transaction and terminates the
	// session; do not retry it.
	Send(ctx context.Context, input RealtimeInput) error
	// Receive returns the events as they arrive from the server. Slow consumers
	// apply backpressure; providers must bound frame sizes and any pending events
	// rather than queue indefinitely. Unknown wire events may be skipped or returned
	// as RealtimeEventProvider events; malformed recognized events are errors. Yielded
	// content must satisfy RealtimeContent.Validate. The sequence stops after the
	// first error, including io.EOF for normal peer closure. Breaking out stops
	// receiving without closing the session. Canceling ctx terminates the transport.
	Receive(ctx context.Context) iter.Seq2[RealtimeEvent, error]
}

RealtimeSession is one provider-owned conversation and its transport.

One goroutine receives events while other goroutines may call Send. Concurrent sends are serialized; callers must order dependent commands themselves. Close may run concurrently and wakes blocked Send and Receive calls. Call it once. The session does not reconnect, replay, execute tools, detect voice activity, or resample.

Example (ManualTurns) ΒΆ
package main

import (
	"context"
	"fmt"
	"log"
	"os"

	"github.com/maruel/genai"
	"github.com/maruel/genai/providers/llamacpp"
)

func main() {
	ctx := context.Background()
	// Run llama-server with a native audio model. The loaded model owns the
	// input and output sample rates. See the llamacppsrv package for a managed
	// runtime.
	c, err := llamacpp.New(ctx, genai.ProviderOptionRemote("http://127.0.0.1:8080"))
	if err != nil {
		log.Print(err)
		return
	}
	defer func() {
		if err := c.Close(); err != nil {
			log.Printf("Close: %v", err)
		}
	}()
	s, err := c.ConnectRealtime(ctx, &llamacpp.RealtimeOption{MaxOutputTokens: 256})
	if err != nil {
		log.Print(err)
		return
	}
	defer func() {
		if err := s.Close(); err != nil {
			log.Printf("Close: %v", err)
		}
	}()

	// A live application captures the microphone at s.Info().InputFormat.
	// This recorded turn is mono PCM16LE at the model's native input rate.
	pcm, err := os.ReadFile("question-16k.pcm")
	if err != nil {
		log.Print(err)
		return
	}
	if err := s.Send(ctx, genai.RealtimeInputContent{Audio: pcm, Commit: true, Generate: true}); err != nil {
		log.Print(err)
		return
	}

	// GenerationDone is not terminal: the provider may emit filler output or a
	// tool call before TurnDone. Keep receiving until TurnDone.
	for e, err := range s.Receive(ctx) {
		if err != nil {
			log.Print(err)
			return
		}
		switch e.Kind {
		case genai.RealtimeEventContentAudio:
			// Play e.Content.Audio at s.Info().OutputFormat.
		case genai.RealtimeEventContentText:
			fmt.Print(e.Content.Text)
		case genai.RealtimeEventContentToolCalls:
			results := make([]genai.ToolCallResult, len(e.Content.ToolCalls))
			for i := range e.Content.ToolCalls {
				results[i] = genai.ToolCallResult{ID: e.Content.ToolCalls[i].ID, Name: e.Content.ToolCalls[i].Name, Result: `{"ok":true}`}
			}
			if err := s.Send(ctx, genai.RealtimeInputContent{ToolResults: results, Generate: true}); err != nil {
				log.Print(err)
				return
			}
		case genai.RealtimeEventTurnDone:
			// TurnDone is terminal: no content or tool call follows it.
			return
		default:
		}
	}
}
Example (ServerTurns) ΒΆ
package main

import (
	"context"
	"fmt"
	"log"

	"github.com/maruel/genai"
	"github.com/maruel/genai/providers/openaichat"
)

func main() {
	ctx := context.Background()
	// llama.cpp supports manual turns only. Use a provider with server turn
	// detection, such as OpenAI Realtime, for this mode.
	c, err := openaichat.New(ctx, genai.ProviderOptionModel("gpt-realtime-2.1"))
	if err != nil {
		log.Print(err)
		return
	}
	defer func() {
		if err := c.Close(); err != nil {
			log.Printf("Close: %v", err)
		}
	}()
	s, err := c.ConnectRealtime(ctx, &genai.RealtimeOptionAudio{OutputTranscription: true})
	if err != nil {
		log.Print(err)
		return
	}
	defer func() {
		if err := s.Close(); err != nil {
			log.Printf("Close: %v", err)
		}
	}()

	// Send a text turn. A microphone loop would send Audio at
	// s.Info().InputFormat instead and let the provider detect the turn end.
	if err := s.Send(ctx, genai.RealtimeInputContent{Text: "Say hello in one word.", Generate: true}); err != nil {
		log.Print(err)
		return
	}
	// GenerationDone is not terminal. Continue until TurnDone, which marks the
	// end of the turn and is never followed by content or tool calls.
	for e, err := range s.Receive(ctx) {
		if err != nil {
			log.Print(err)
			return
		}
		switch e.Kind {
		case genai.RealtimeEventContentAudio:
			// Play e.Content.Audio at s.Info().OutputFormat.
		case genai.RealtimeEventTranscriptOutput:
			fmt.Print(e.Content.Text)
		case genai.RealtimeEventError:
			log.Printf("Provider rejected a command: %v", e.Error)
		case genai.RealtimeEventInterrupted:
			// Stop playback and discard queued audio.
		case genai.RealtimeEventTurnDone:
			// TurnDone is terminal: no content or tool call follows it.
			return
		default:
		}
	}
}

type RealtimeSessionInfo ΒΆ added in v0.11.0

type RealtimeSessionInfo struct {
	ID           string
	InputFormat  AudioFormat
	OutputFormat AudioFormat
	// InputModalities lists the payload kinds the session encodes: ModalityAudio,
	// ModalityImage, and ModalityText. It states what the transport accepts.
	InputModalities []Modality
	// TurnDetection is the accepted server or manual turn mode.
	TurnDetection RealtimeTurnDetection
	Capabilities  RealtimeCapabilities
}

RealtimeSessionInfo contains the accepted session identity, native formats, accepted input kinds, turn mode, and capabilities. IDs are provider-supplied when available; an empty ID means none was supplied.

type RealtimeToolCallCancelledError ΒΆ added in v0.11.0

type RealtimeToolCallCancelledError struct {
	// ToolCallIDs are the cancelled calls matched in the rejected batch.
	ToolCallIDs []string
}

RealtimeToolCallCancelledError means a tool result batch matched one or more calls the provider cancelled. The batch was rejected locally without wire effects. Callers may discard the listed calls and retry the remaining results.

func (*RealtimeToolCallCancelledError) Error ΒΆ added in v0.11.0

Error implements error.

type RealtimeTruncate ΒΆ added in v0.11.0

type RealtimeTruncate struct {
	ItemID       string
	ContentIndex int
	AudioEndMS   int64
}

RealtimeTruncate reconciles unheard output audio in ItemID at ContentIndex and AudioEndMS. It requires Capabilities.Truncate and does not stop generation.

func (RealtimeTruncate) Validate ΒΆ added in v0.11.0

func (i RealtimeTruncate) Validate() error

Validate checks the truncation target.

type RealtimeTurnDetection ΒΆ added in v0.11.0

type RealtimeTurnDetection string

RealtimeTurnDetection selects who decides that microphone input is complete.

const (
	RealtimeTurnManual RealtimeTurnDetection = "manual"
	RealtimeTurnServer RealtimeTurnDetection = "server"
)

RealtimeTurnDetection values. Empty selects the provider's documented default.

func (RealtimeTurnDetection) Validate ΒΆ added in v0.11.0

func (o RealtimeTurnDetection) Validate() error

Validate implements RealtimeOption.

type RealtimeUpdate ΒΆ added in v0.11.0

type RealtimeUpdate struct {
	Instructions     *string
	Tools            *[]ToolDef
	ToolChoice       *ToolCallRequest
	MaxOutputTokens  *int64
	OutputModalities *Modalities
	TurnDetection    *RealtimeTurnDetection
	Voice            *string
}

RealtimeUpdate changes live session settings. A nil field is unchanged. Tools may be an empty list to clear declarations. Instructions may be empty to clear them. MaxOutputTokens must be positive. TurnDetection selects manual or server turns. Voice changes only before output audio on providers that report VoiceBeforeOutput.

func (RealtimeUpdate) Validate ΒΆ added in v0.11.0

func (u RealtimeUpdate) Validate() error

Validate rejects empty, malformed, or ambiguous updates.

type RealtimeUpdateCapabilities ΒΆ added in v0.11.0

type RealtimeUpdateCapabilities struct {
	Instructions       bool
	Tools              bool
	ToolChoice         bool
	ToolChoiceRequired bool
	MaxOutputTokens    bool
	OutputModalities   bool
	TurnDetection      bool
	VoiceBeforeOutput  bool
}

RealtimeUpdateCapabilities describes the session settings this configured session accepts at runtime. ToolChoice covers auto and none; required has its own field because some providers can clear tools but cannot force a call. VoiceBeforeOutput rejects a voice change after the session produced audio.

type ReasoningOutput ΒΆ added in v0.11.0

type ReasoningOutput struct{}

ReasoningOutput requests optional reasoning or summaries for generation and realtime sessions. Omitting it selects the provider's cheapest output mode. Returned reasoning and native continuation state are always preserved.

var IncludeReasoning ReasoningOutput

IncludeReasoning requests optional reasoning where available. It does not enable thinking or guarantee that the model returns reasoning.

func (ReasoningOutput) Validate ΒΆ added in v0.11.0

func (ReasoningOutput) Validate() error

Validate implements GenOption and RealtimeOption.

type Reply ΒΆ

type Reply struct {
	// Text is the content of the text message.
	Text string `json:"text,omitzero"`

	// Doc can be audio, video, image, PDF or any other format, including reference text.
	Doc Doc `json:"doc,omitzero"`

	// Citation contains references to source material that support the content.
	Citation Citation `json:"citation,omitzero"`

	// Reasoning is the reasoning done by the LLM. Keep it in conversation history;
	// some models require it when continuing a turn after tool calls.
	Reasoning string `json:"reasoning,omitzero"`

	// ToolCall is a tool call that the LLM requested to make.
	ToolCall ToolCall `json:"tool_call,omitzero"`

	// Opaque is added to keep continuity on the processing. A good example is Anthropic's extended thinking, or
	// server-side tool calling. It must be kept during an exchange.
	//
	// A message with only Opaque set is valid. It can be used in combination with other fields. This field is
	// specific to both the provider and the model.
	//
	// The data must be JSON-serializable.
	Opaque map[string]any `json:"opaque,omitzero"`
	// contains filtered or unexported fields
}

Reply is a block of information returned by the provider.

Normally only one of the field must be set. The exception is the Opaque field.

Reply generally represents content returned by the provider, like a block of text or a document returned by the model. It can be a silent tool call request. It can also be an opaque block. A good example is traces of server side tool calling like Web search or MCP tool calling.

func (*Reply) GoString ΒΆ

func (r *Reply) GoString() string

GoString returns a JSON representation of the reply for debugging purposes.

func (*Reply) IsZero ΒΆ

func (r *Reply) IsZero() bool

IsZero returns true if the Reply is empty.

An empty reply is not valid.

func (*Reply) UnmarshalJSON ΒΆ

func (r *Reply) UnmarshalJSON(b []byte) error

UnmarshalJSON implements the json.Unmarshaler interface.

func (*Reply) Validate ΒΆ

func (r *Reply) Validate() error

Validate ensures the block is valid.

type Request ΒΆ

type Request struct {
	// Text is the content of the text message.
	Text string `json:"text,omitzero"`

	// Doc can be audio, video, image, PDF or any other format, including reference text.
	Doc Doc `json:"doc,omitzero"`
	// contains filtered or unexported fields
}

Request is an input content block for generation or embedding.

It is effectively a union, only one of the 2 related field groups can be set.

func (*Request) UnmarshalJSON ΒΆ

func (r *Request) UnmarshalJSON(b []byte) error

UnmarshalJSON implements the json.Unmarshaler interface.

func (*Request) Validate ΒΆ

func (r *Request) Validate() error

Validate ensures the block is valid.

type Result ΒΆ

type Result struct {
	Message

	Usage Usage
	// Logprobs is a list of multiple log probabilities, each for a token.
	//
	// The first item of each subslice is the chosen token. The next items are the candidates not chosen.
	//
	// Some providers only return the probability for the chosen tokens and not for the candidates.
	Logprobs [][]Logprob
}

Result is the result of a completion.

It is a Message along with Usage metadata about the operation. It optionally include Logprobs if requested and the provider support it.

func (*Result) Validate ΒΆ added in v0.4.0

func (r *Result) Validate() error

Validate ensures the result is valid.

type ScoreLegend ΒΆ added in v0.10.0

type ScoreLegend map[string]DecisionContent

ScoreLegend is the description of each level of a score answer, keyed by the level.

It is the Question.Score criteria of the question, echoed back by the API. The values are decoded as Text, Object or Array; a nil value is a level the caller left undescribed.

func (*ScoreLegend) UnmarshalJSON ΒΆ added in v0.10.0

func (l *ScoreLegend) UnmarshalJSON(b []byte) error

UnmarshalJSON implements json.Unmarshaler.

DecisionContent is an interface, so encoding/json cannot decode it on its own; each level is decoded into the variant matching its JSON token here.

type ScoreboardVariant ΒΆ added in v0.3.0

type ScoreboardVariant struct {
	// Name is the display name for this variant, e.g. "Intl", "US".
	Name string
	// Score is the scoreboard data for this variant.
	Score scoreboard.Score
}

ScoreboardVariant is a named scoreboard for a specific backend or region of a provider.

type Starter ΒΆ added in v0.4.1

type Starter func(ctx context.Context, args []string) (stdin io.WriteCloser, stdout io.ReadCloser, wait func() error, err error)

Starter launches a subprocess and returns its stdin, stdout, a wait function, and any error.

It is the subprocess equivalent of http.RoundTripper: the lowest-level abstraction that CLI-based providers (claudecode, codex, opencode) use to spawn a process.

type SystemOneRequest ΒΆ added in v0.10.0

type SystemOneRequest struct {
	// State is required unless Docs contains one application/json document.
	State     DecisionContent `json:"state"`
	Questions Questions       `json:"questions"`
	// Docs contains inline document attachments. When State is unset, one application/json document
	// supplies the state instead. Supported attachment types depend on the provider and model.
	// Remote URLs are not supported.
	Docs []Doc `json:"docs,omitzero"`
}

SystemOneRequest is a System One decision request. Providers use their client's configured model and own transport envelopes and stricter protocol limits. Docs are inline attachments. Providers select supported document types, enforce read limits, and encode document bytes in their native wire format.

func (*SystemOneRequest) ReadState ΒΆ added in v0.10.0

func (r *SystemOneRequest) ReadState(maxDocBytes int64) (DecisionContent, []Doc, error)

ReadState validates the request and returns the resolved state and remaining attachments. It reads a JSON state document with maxDocBytes as the byte limit, but does not read attachments. It leaves request fields unchanged. Returned attachments share the request's Docs slice.

func (*SystemOneRequest) Validate ΒΆ added in v0.10.0

func (r *SystemOneRequest) Validate() error

Validate checks the shared state, questions and inline documents without reading attachment bytes. Providers validate document types.

type SystemOneResponse ΒΆ added in v0.10.0

type SystemOneResponse struct {
	// Answers holds one answer per question, keyed by the question name.
	Answers Answers `json:"answers"`
	// Usage is the token usage of the request.
	Usage DecisionUsage `json:"usage"`
}

SystemOneResponse is the response of a POST /v1/systemone request.

func (*SystemOneResponse) ValidateQuestions ΒΆ added in v0.10.0

func (r *SystemOneResponse) ValidateQuestions(qs *Questions) error

ValidateQuestions checks that the response answers each requested question with the right type.

type Text ΒΆ added in v0.10.0

type Text string

Text is DecisionContent that is a JSON string.

func (Text) Validate ΒΆ added in v0.10.0

func (Text) Validate() error

Validate implements internal.Validatable.

Any text is valid, including the empty string, the API is the one that decides whether a specific field may be empty.

type ToolCall ΒΆ

type ToolCall struct {
	ID        string `json:"id,omitzero"`        // Unique identifier for the tool call. Necessary for parallel tool calling.
	Name      string `json:"name,omitzero"`      // Tool being called.
	Arguments string `json:"arguments,omitzero"` // encoded as JSON

	// Opaque is added to keep continuity on the processing. A good example is Anthropic's extended thinking. It
	// must be kept during an exchange.
	//
	// A message with only Opaque set is valid.
	Opaque map[string]any `json:"opaque,omitzero"`
	// contains filtered or unexported fields
}

ToolCall is a tool call that the LLM requested to make.

func (*ToolCall) Call ΒΆ

func (t *ToolCall) Call(ctx context.Context, tools []ToolDef) (string, error)

Call invokes the ToolDef.Callback with arguments from the ToolCall, returning the result string.

It decodes the ToolCall.Arguments and passes it to the ToolDef.Callback.

func (*ToolCall) IsZero ΒΆ

func (t *ToolCall) IsZero() bool

IsZero returns true if the tool call is empty.

func (*ToolCall) UnmarshalJSON ΒΆ

func (t *ToolCall) UnmarshalJSON(b []byte) error

UnmarshalJSON implements the json.Unmarshaler interface.

func (*ToolCall) Validate ΒΆ

func (t *ToolCall) Validate() error

Validate ensures the tool call request from the LLM is valid.

type ToolCallRequest ΒΆ

type ToolCallRequest int

ToolCallRequest determines if we want the LLM to request a tool call.

const (
	// ToolCallAny is the default, the model is free to choose if a tool is called or not. For some models (like
	// llama family), it may be a bit too "tool call happy".
	ToolCallAny ToolCallRequest = iota
	// ToolCallRequired means a tool call is required. Don't forget to change the value after sending the
	// response!
	ToolCallRequired
	// ToolCallNone means that while tools are described, they should not be called. It is useful when a LLM did
	// tool calls, got the response and now it's time to generate some text to present to the end user.
	ToolCallNone
)

type ToolCallResult ΒΆ

type ToolCallResult struct {
	ID     string `json:"id,omitzero"`
	Name   string `json:"name,omitzero"`
	Result string `json:"result,omitzero"`
	// contains filtered or unexported fields
}

ToolCallResult is the result for a tool call that the LLM requested to make.

func (*ToolCallResult) UnmarshalJSON ΒΆ

func (t *ToolCallResult) UnmarshalJSON(b []byte) error

UnmarshalJSON implements the json.Unmarshaler interface.

func (*ToolCallResult) Validate ΒΆ

func (t *ToolCallResult) Validate() error

Validate ensures the tool result is valid.

type ToolDef ΒΆ

type ToolDef struct {
	// Name must be unique among all tools.
	Name string
	// Description must be a LLM-friendly short description of the tool.
	Description string
	// Callback is the function to call with the inputs.
	// It must accept a context.Context one struct pointer as input: (ctx context.Context, input *struct{}). The
	// struct must use json_schema to be serializable as JSON.
	// It must return the result and an error: (string, error).
	Callback any
	// InputSchemaOverride overrides the schema deduced from the Callback's second argument. It's meant to be
	// used when an enum or a description is set dynamically, or with complex if/then/else that would be tedious
	// to describe as struct tags.
	//
	// It is okay to initialize Callback, then take the return value of GetInputSchema() to initialize InputSchemaOverride, then mutate it.
	InputSchemaOverride JSONSchema
	// contains filtered or unexported fields
}

ToolDef describes a tool that the LLM can request to use.

func (*ToolDef) GetInputSchema ΒΆ

func (t *ToolDef) GetInputSchema() (JSONSchema, error)

GetInputSchema returns the json schema for the input argument of the callback.

func (*ToolDef) Validate ΒΆ

func (t *ToolDef) Validate() error

Validate ensures the tool definition is valid.

For the Name field, it uses the rule according to https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/implement-tool-use#example-simple-tool-definition

type Usage ΒΆ

type Usage struct {
	// Token usage for the current request.
	InputTokens       int64
	InputCachedTokens int64
	ReasoningTokens   int64
	OutputTokens      int64
	TotalTokens       int64
	// ContextTokens is the number of tokens in the context window after the
	// request. Zero when the provider does not report it.
	ContextTokens int64
	// ContextWindow is the maximum number of tokens the context window holds.
	// Zero when the provider does not report it.
	ContextWindow int64
	// FinishReason indicates why the model stopped generating tokens.
	FinishReason FinishReason
	// ServiceTier is the quality of service tier used to process the request,
	// as reported by the provider. Common values include "standard", "batch",
	// "flex", "default", "auto", etc. Empty when not reported.
	ServiceTier string
	// Limits contains a list of rate limit details from the provider.
	Limits []RateLimit
}

Usage from the LLM provider.

func (*Usage) Add ΒΆ

func (u *Usage) Add(r *Usage)

Add accumulates the token counts from another result. It does not accumulate the context fields, which are a snapshot of the context window.

func (*Usage) String ΒΆ

func (u *Usage) String() string

Directories ΒΆ

Path Synopsis
Package acp implements a client for the Agent Client Protocol (ACP).
Package acp implements a client for the Agent Client Protocol (ACP).
Package adapters provides adapters for genai providers and realtime sessions.
Package adapters provides adapters for genai providers and realtime sessions.
Package base provides shared infrastructure for implementing genai providers.
Package base provides shared infrastructure for implementing genai providers.
cmd
cache-mgr command
Command cache-mgr fetches and prints out the list of files stored on the selected provider.
Command cache-mgr fetches and prints out the list of files stored on the selected provider.
list-models command
Command list-models fetches and prints out the list of models from the selected providers.
Command list-models fetches and prints out the list of models from the selected providers.
llama-serve command
Command llama-serve fetches a model from HuggingFace and runs llama-server.
Command llama-serve fetches a model from HuggingFace and runs llama-server.
scoreboard command
Command scoreboard generates a scoreboard for every providers supported.
Command scoreboard generates a scoreboard for every providers supported.
examples
aud-txt_to_txt command
img-txt_to_img command
img-txt_to_txt command
img-txt_to_vid command
txt_to_img command
txt_to_txt_any command
txt_to_txt_oauth2_codex command
Authenticate with Codex's saved ChatGPT subscription and call GPT-6 Luna directly over HTTP, without launching Codex.
Authenticate with Codex's saved ChatGPT subscription and call GPT-6 Luna directly over HTTP, without launching Codex.
txt_to_txt_sync command
vid-txt_to_txt command
Package httprecord provides safe HTTP recording logic for users that was to understand the API and do smoke tests.
Package httprecord provides safe HTTP recording logic for users that was to understand the API and do smoke tests.
Package internal provides shared utilities for the genai library.
Package internal provides shared utilities for the genai library.
bb
Package bb is a separate package so it can be imported by genai while being internal and exported so cmp.Diff() isn't unhappy.
Package bb is a separate package so it can be imported by genai while being internal and exported so cmp.Diff() isn't unhappy.
cli
Package cli provides process helpers for CLI-based providers.
Package cli provides process helpers for CLI-based providers.
cmd/autofix-weekly-regen command
Command autofix-weekly-regen asks pi.dev to repair weekly model regeneration failures.
Command autofix-weekly-regen asks pi.dev to repair weekly model regeneration failures.
cmd/gen-realtimespeech command
Command gen-realtimespeech regenerates the realtime speech FLAC fixtures with llama-tts.
Command gen-realtimespeech regenerates the realtime speech FLAC fixtures with llama-tts.
cmd/update-servers command
Command update-servers checks for newer releases of llama.cpp and ollama servers used by llamacppsrv and ollamasrv, updates the version constants, and regenerates test recordings.
Command update-servers checks for newer releases of llama.cpp and ollama servers used by llamacppsrv and ollamasrv, updates the version constants, and regenerates test recordings.
ghrelease
Package ghrelease provides shared helpers for downloading and extracting GitHub release assets.
Package ghrelease provides shared helpers for downloading and extracting GitHub release assets.
internaltest
Package internaltest is awesome sauce for unit testing.
Package internaltest is awesome sauce for unit testing.
msgutil
Package msgutil provides shared message utilities for subprocess-based providers.
Package msgutil provides shared message utilities for subprocess-based providers.
myrecorder
Package myrecorder has HTTP recording logic.
Package myrecorder has HTTP recording logic.
realtime
Package realtime provides cancellable WebSocket I/O with bounded reads for realtime providers.
Package realtime provides cancellable WebSocket I/O with bounded reads for realtime providers.
realtimetest
Package realtimetest provides scripted WebSocket peers, session checks and a fake session.
Package realtimetest provides scripted WebSocket peers, session checks and a fake session.
sse
Package sse provides Server-Sent Events (SSE) processing utilities.
Package sse provides Server-Sent Events (SSE) processing utilities.
Package providers is the root of all standard providers.
Package providers is the root of all standard providers.
alibaba
Package alibaba implements a client for the Alibaba Cloud DashScope API (Model Studio).
Package alibaba implements a client for the Alibaba Cloud DashScope API (Model Studio).
anthropic
Package anthropic implements a client for the Anthropic API, to use Claude.
Package anthropic implements a client for the Anthropic API, to use Claude.
antigravity
Package antigravity implements a genai provider backed by the Google Antigravity CLI (`agy`).
Package antigravity implements a genai provider backed by the Google Antigravity CLI (`agy`).
antigravity/internal/extracttypes command
Command extracttypes prints the struct types, with field tags, embedded in a stripped Go binary.
Command extracttypes prints the struct types, with field tags, embedded in a stripped Go binary.
baseten
Package baseten implements a client for the Baseten inference API.
Package baseten implements a client for the Baseten inference API.
bfl
Package bfl implements a client for Black Forest Labs API.
Package bfl implements a client for Black Forest Labs API.
cerebras
Package cerebras implements a client for the Cerebras API.
Package cerebras implements a client for the Cerebras API.
claudecode
Package claudecode implements a genai provider backed by the Claude Code CLI.
Package claudecode implements a genai provider backed by the Claude Code CLI.
cloudflare
Package cloudflare implements a client for the Cloudflare AI API.
Package cloudflare implements a client for the Cloudflare AI API.
codex
Package codex implements a genai provider backed by the Codex CLI.
Package codex implements a genai provider backed by the Codex CLI.
cohere
Package cohere implements a client for the Cohere API.
Package cohere implements a client for the Cohere API.
deepseek
Package deepseek implements a client for the DeepSeek API.
Package deepseek implements a client for the DeepSeek API.
dsh
Package dsh implements a genai provider backed by the DeepSeek Harness CLI.
Package dsh implements a genai provider backed by the DeepSeek Harness CLI.
gemini
Package gemini implements a client for Google's Gemini API.
Package gemini implements a client for Google's Gemini API.
groq
Package groq implements a client for the Groq API.
Package groq implements a client for the Groq API.
huggingface
Package huggingface implements a client for the HuggingFace serverless inference API.
Package huggingface implements a client for the HuggingFace serverless inference API.
llamacpp
Package llamacpp implements a client for llama-server native APIs.
Package llamacpp implements a client for llama-server native APIs.
llamacpp/llamacppsrv
Package llamacppsrv downloads llama.cpp executables and starts llama-server.
Package llamacppsrv downloads llama.cpp executables and starts llama-server.
mistral
Package mistral implements a client for the Mistral API.
Package mistral implements a client for the Mistral API.
ollama
Package ollama implements a client for the Ollama API.
Package ollama implements a client for the Ollama API.
ollama/ollamasrv
Package ollamasrv downloads and starts ollama directly from GitHub releases.
Package ollamasrv downloads and starts ollama directly from GitHub releases.
openaibase
Package openaibase contains shared types and client operations used by the OpenAI Chat Completion, Responses and Realtime providers, and by compatible providers such as llama.cpp.
Package openaibase contains shared types and client operations used by the OpenAI Chat Completion, Responses and Realtime providers, and by compatible providers such as llama.cpp.
openaichat
Package openaichat implements a client for the OpenAI Chat Completion API.
Package openaichat implements a client for the OpenAI Chat Completion API.
openaicompatible
Package openaicompatible implements a lenient client for "OpenAI-compatible" providers.
Package openaicompatible implements a lenient client for "OpenAI-compatible" providers.
openairesponses
Package openairesponses implements a client for the OpenAI Responses API.
Package openairesponses implements a client for the OpenAI Responses API.
opencode
Package opencode implements a genai provider backed by the OpenCode CLI.
Package opencode implements a genai provider backed by the OpenCode CLI.
openrouter
Package openrouter implements a client for the OpenRouter API.
Package openrouter implements a client for the OpenRouter API.
pi
Package pi implements a genai provider backed by the Pi coding agent CLI.
Package pi implements a genai provider backed by the Pi coding agent CLI.
pollinations
Package pollinations implements a client for the Pollinations API.
Package pollinations implements a client for the Pollinations API.
togetherai
Package togetherai implements a client for the Together.ai API.
Package togetherai implements a client for the Together.ai API.
typesafe
Package typesafe implements a client for the TypeSafe System One API, to use Jev.
Package typesafe implements a client for the TypeSafe System One API, to use Jev.
xiaomi
Package xiaomi implements a client for the Xiaomi MiMo platform API.
Package xiaomi implements a client for the Xiaomi MiMo platform API.
Package scoreboard declares the structures to define a scoreboard.
Package scoreboard declares the structures to define a scoreboard.
Package smoke defines probe families and the recorder they measure through.
Package smoke defines probe families and the recorder they measure through.
smoketest
Package smoketest runs a scoreboard in test mode.
Package smoketest runs a scoreboard in test mode.
Package subprocessrecord provides recording and replay of subprocess I/O for CLI-based providers (claudecode, codex, dsh, opencode).
Package subprocessrecord provides recording and replay of subprocess I/O for CLI-based providers (claudecode, codex, dsh, opencode).
Package websocketrecord provides recording and replay of WebSocket message exchanges for provider smoke tests.
Package websocketrecord provides recording and replay of WebSocket message exchanges for provider smoke tests.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL