voicegemini

package
v0.2.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 23, 2026 License: MIT Imports: 10 Imported by: 0

Documentation

Overview

Package voicegemini implements the Gemini Live half of pi's web voice session: key/model verification, the BidiGenerateContent wire shapes, and the setup message the server-side relay opens every session with.

The transport is a server-side relay, not a browser-to-provider connection. The Gemini Live ephemeral-token flow is SDK-documented only (its REST shape is not published), so pi's web server relays: the browser's only peer is this server's WebSocket endpoint, and the server dials the Live API with the long-lived GEMINI_API_KEY. The key therefore never leaves the server in any form — there is no ephemeral credential to mint, scope, or leak.

Wire protocol verified against ai.google.dev/api/live and the Live API guide on 2026-08-14: WebSocket wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent, client envelopes setup | clientContent | realtimeInput | toolResponse, server envelopes setupComplete | serverContent | toolCall | toolCallCancellation | goAway; input audio is raw little-endian 16-bit PCM at 16kHz ("audio/pcm;rate=16000"), output audio is 24kHz PCM. Re-verify before trusting this code — the Live API is preview-tier and moves.

Index

Constants

View Source
const DefaultModel = "gemini-3.1-flash-live-preview"

DefaultModel is the Live API native-audio model. Override with GEMINI_LIVE_MODEL when the account has a newer one; Verify checks whatever is configured against the key, and the override deliberately bypasses LiveModels (see there).

It is itself a preview, and a dated preview being retired is the usual way this constant goes stale. That failure mode is survivable because GEMINI_LIVE_MODEL repoints the server at any Live model with no code change, and because Verify turns a retirement into a named boot error instead of a dead microphone.

View Source
const InputMimeType = "audio/pcm;rate=16000"

InputMimeType is the realtimeInput audio format the relay forwards.

View Source
const InputSampleRate = 16000

InputSampleRate is the rate the relay expects mic audio at, matching InputMimeType.

View Source
const LiveMethod = "bidiGenerateContent"

LiveMethod is the generation method a Live (BidiGenerateContent) session needs. Most Gemini models exist and answer GET /models/<name> while having no Live support at all, so "the model exists" is not the check worth making.

View Source
const LiveURL = "wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent"

LiveURL is the BidiGenerateContent WebSocket endpoint. The API key is passed as a query parameter on dial, the way the GenAI SDKs do it.

View Source
const OutputSampleRate = 24000

OutputSampleRate is the rate the Live API streams output audio at. The browser builds its playback AudioContext at this rate; it is here so the server and the page cannot drift apart silently.

View Source
const SessionTTL = 30 * time.Minute

SessionTTL bounds how long a created session descriptor stays usable before the browser must ask for a new one.

Variables

View Source
var LiveModels = []string{
	DefaultModel,
}

LiveModels are the models this build offers the *browser* for per-session selection. It is an allowlist against a browser naming an arbitrary model, not a catalog of what works — GEMINI_LIVE_MODEL bypasses it entirely, because an operator setting an env var on their own server is a different trust level from a page choosing a model, and Verify still gates whatever they picked. Models keeps such an override selectable so switching away from it is not a one-way door.

Re-probe before adding to this list. Most models the key advertises do not serve bidiGenerateContent at all, and some that do are not assistants — a translate-preview model never emits a toolCall or a turnComplete, so offering it would fail at the microphone rather than at startup.

Functions

func MaskKey

func MaskKey(k string) string

MaskKey renders a key as head***tail plus its length — enough to tell two keys apart in a log (a stale shell export vs the one in the environment) without ever printing a usable credential. Short strings are shown as *** only.

func RealtimeAudioMessage

func RealtimeAudioMessage(b64 string) map[string]any

RealtimeAudioMessage wraps one chunk of raw PCM16@16k mic audio (already base64-encoded) as a realtimeInput client message.

func SanitizeSchema

func SanitizeSchema(raw json.RawMessage) (json.RawMessage, error)

SanitizeSchema rewrites a JSON Schema into the OpenAPI-3.0 subset that google.ai.generativelanguage.v1beta.Schema accepts. That message is parsed by a strict proto-JSON parser: any field it does not know closes the live session with WebSocket 1007 at setup, before a single microphone byte is sent.

Two constructs Go schema inference commonly emits are fatal there:

  • "additionalProperties" — emitted for every struct; the field does not exist in Gemini's Schema at all. Dropped at every depth. Nothing is lost that Gemini could have enforced, and unknown arguments are still rejected server-side when the call is decoded.
  • an array-valued "type" — emitted for every pointer or slice field ("type": ["null","array"]); Gemini's type is a single enum. Collapsed to its non-null member, with "nullable": true when "null" was present.

It is deliberately permissive and total: it strips only what it knows about and passes everything else through, including input that is not a JSON object at all. A keyword Gemini would reject but this function does not know must fail a guard test, not disable voice at startup.

It operates on marshaled bytes and returns fresh bytes, so it can never mutate a schema value shared with pi's other tool paths.

func ToolResponseMessage

func ToolResponseMessage(responses []FunctionResponse) map[string]any

ToolResponseMessage answers one or more provider function calls.

Types

type Content

type Content struct {
	Role  string `json:"role,omitempty"`
	Parts []Part `json:"parts,omitempty"`
}

Content is a minimal GenAI content: role + parts.

type Creator

type Creator struct {
	APIKey       string
	Model        string
	Instructions string
	// InstructionsFunc, when set, is consulted when the relay opens a session
	// (SetupMessage) and wins over Instructions unless it returns empty. It is
	// the hook for an instruction that depends on live server state.
	InstructionsFunc func() string
	Tools            []Tool

	ModelsURL  string // override for tests; empty means the real endpoint
	LiveURL    string // override for tests; empty means the real endpoint
	HTTPClient *http.Client
}

Creator holds the server-side Gemini Live configuration. It implements voice.SessionCreator, but does no provider round-trip per session: the relay dials the provider when the browser connects, so Create only shapes the transport descriptor the browser needs.

func New

func New(apiKey string, opts ...Option) *Creator

New returns a Creator for the given long-lived API key.

func (*Creator) Create

func (c *Creator) Create(_ context.Context, _ string) (voice.Session, error)

Create implements voice.SessionCreator. The Realtime map tells the browser to use pi's relay transport; it carries no credential at all, because the relay authenticates by web session and dials the provider itself.

func (*Creator) DialURL

func (c *Creator) DialURL() string

DialURL is the provider WebSocket URL including the key. Only the server ever sees this string — never log it.

func (*Creator) Models

func (c *Creator) Models() []string

Models is what this Creator offers the browser to choose from: the verified allowlist, plus the configured model when an operator pinned something outside it. An operator override that could not be selected back after switching away would be a trap.

func (*Creator) SetupMessage

func (c *Creator) SetupMessage() map[string]any

SetupMessage builds the first client message of a session.

func (*Creator) Verify

func (c *Creator) Verify(ctx context.Context) error

Verify checks the key and the model against the models endpoint: the key is accepted, the model exists, and the model actually serves the Live API. An invalid key fails here with the provider's own words instead of surfacing later as a dial failure only the server log would see.

The Live check is the one that matters at the microphone. A model that exists but lacks bidiGenerateContent completes the HTTP verification and then kills every session mid-conversation with a WebSocket 1007 the browser has to explain — so verification asks the models endpoint what the model can do, not merely whether it is there.

func (*Creator) WithModelSelection

func (c *Creator) WithModelSelection(sel string) *Creator

WithModelSelection returns a shallow copy of c bound to one browser-selected model, or c itself when the selection is empty, unknown, or already current.

The copy is the point: two tabs must be able to run different models at once, so a per-session choice must never mutate the shared Creator. So is the allowlist check — SetupMessage's contract is that the browser cannot choose the model, and that survives here because the browser picks an entry from a server-defined list rather than supplying a model string. A stale tab naming a model this build no longer offers falls back to the default instead of failing the session.

func (*Creator) WithSessionInstructions

func (c *Creator) WithSessionInstructions(text string) *Creator

WithSessionInstructions returns a copy of c whose system instruction is the given text, or c itself when the text is empty.

It is always a copy, never a mutation, for the same reason WithModelSelection is: two tabs run two live sessions off one shared Creator, and a per-session instruction — which is what an instruction carrying the current terminal's contents necessarily is — must not become every session's instruction. Empty is a deliberate no-op so a caller that has nothing session-specific to say falls through to whatever WithInstructions or InstructionsFunc configured.

type FunctionCall

type FunctionCall struct {
	ID   string          `json:"id,omitempty"`
	Name string          `json:"name"`
	Args json.RawMessage `json:"args,omitempty"`
}

FunctionCall is one requested call.

type FunctionResponse

type FunctionResponse struct {
	ID       string         `json:"id,omitempty"`
	Name     string         `json:"name"`
	Response map[string]any `json:"response"`
}

FunctionResponse is one function result, matched to its call by ID.

type GenerationConfig

type GenerationConfig struct {
	ResponseModalities []string `json:"responseModalities,omitempty"`
}

GenerationConfig carries the response modalities the session asks for.

type GoAway

type GoAway struct {
	TimeLeft string `json:"timeLeft,omitempty"`
}

GoAway warns that the provider is about to end the session.

type InlineData

type InlineData struct {
	MimeType string `json:"mimeType"`
	Data     string `json:"data"` // base64
}

InlineData is a base64 blob with its MIME type.

type Option

type Option func(*Creator)

Option configures a Creator.

func WithHTTPClient

func WithHTTPClient(h *http.Client) Option

WithHTTPClient overrides the HTTP client (tests only).

func WithInstructions

func WithInstructions(i string) Option

WithInstructions sets the live-session system instructions.

func WithInstructionsFunc

func WithInstructionsFunc(f func() string) Option

WithInstructionsFunc sets a per-session instructions source (see the field).

func WithLiveURL

func WithLiveURL(u string) Option

WithLiveURL overrides the Live WebSocket endpoint (tests only).

func WithModel

func WithModel(m string) Option

WithModel pins the Live model, bypassing the LiveModels allowlist.

func WithModelsURL

func WithModelsURL(u string) Option

WithModelsURL overrides the models endpoint (tests only).

func WithTools

func WithTools(t []Tool) Option

WithTools sets the function declarations exposed to the live session.

type Part

type Part struct {
	Text       string      `json:"text,omitempty"`
	InlineData *InlineData `json:"inlineData,omitempty"`
}

Part is one content part: text or inline binary data.

type ServerContent

type ServerContent struct {
	ModelTurn      *Content       `json:"modelTurn,omitempty"`
	Interrupted    bool           `json:"interrupted,omitempty"`
	TurnComplete   bool           `json:"turnComplete,omitempty"`
	InputTranscr   *Transcription `json:"inputTranscription,omitempty"`
	OutputTranscr  *Transcription `json:"outputTranscription,omitempty"`
	GenerationDone bool           `json:"generationComplete,omitempty"`
}

ServerContent is the model's side of one turn.

func (*ServerContent) AudioParts

func (sc *ServerContent) AudioParts() []string

AudioParts returns the base64 PCM payloads of a model turn, in order. The Live API streams output audio as inlineData parts at OutputSampleRate.

type ServerMessage

type ServerMessage struct {
	SetupComplete *struct{}      `json:"setupComplete,omitempty"`
	ServerContent *ServerContent `json:"serverContent,omitempty"`
	ToolCall      *ToolCall      `json:"toolCall,omitempty"`
	ToolCancel    *ToolCancel    `json:"toolCallCancellation,omitempty"`
	GoAway        *GoAway        `json:"goAway,omitempty"`
}

ServerMessage is one BidiGenerateContentServerMessage. Exactly one of the pointer fields is set.

type Setup

type Setup struct {
	Model             string             `json:"model"`
	GenerationConfig  *GenerationConfig  `json:"generationConfig,omitempty"`
	SystemInstruction *Content           `json:"systemInstruction,omitempty"`
	Tools             []ToolDeclarations `json:"tools,omitempty"`
	InputTranscribe   *struct{}          `json:"inputAudioTranscription,omitempty"`
	OutputTranscribe  *struct{}          `json:"outputAudioTranscription,omitempty"`
}

Setup is the BidiGenerateContentSetup message the relay sends first. The server owns it entirely — the browser cannot choose the model, instructions, or tools.

type Tool

type Tool struct {
	Name        string          `json:"name"`
	Description string          `json:"description"`
	Parameters  json.RawMessage `json:"parameters,omitempty"`
}

Tool mirrors the GenAI functionDeclarations shape for one exposed capability.

type ToolCall

type ToolCall struct {
	FunctionCalls []FunctionCall `json:"functionCalls,omitempty"`
}

ToolCall carries the function calls the model wants executed.

type ToolCancel

type ToolCancel struct {
	IDs []string `json:"ids,omitempty"`
}

ToolCancel names calls the model no longer wants answered.

type ToolDeclarations

type ToolDeclarations struct {
	FunctionDeclarations []Tool `json:"functionDeclarations,omitempty"`
}

ToolDeclarations wraps the function declarations for one tools entry.

type Transcription

type Transcription struct {
	Text string `json:"text,omitempty"`
}

Transcription is one transcript fragment. Gemini streams true increments for both directions, not whole utterances.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL