Documentation
¶
Overview ¶
Package voicegemini implements the Gemini Live half of pi's web voice session: key/model verification, the BidiGenerateContent wire shapes, and the setup message the server-side relay opens every session with.
The transport is a server-side relay, not a browser-to-provider connection. The Gemini Live ephemeral-token flow is SDK-documented only (its REST shape is not published), so pi's web server relays: the browser's only peer is this server's WebSocket endpoint, and the server dials the Live API with the long-lived GEMINI_API_KEY. The key therefore never leaves the server in any form — there is no ephemeral credential to mint, scope, or leak.
Wire protocol verified against ai.google.dev/api/live and the Live API guide on 2026-08-14: WebSocket wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent, client envelopes setup | clientContent | realtimeInput | toolResponse, server envelopes setupComplete | serverContent | toolCall | toolCallCancellation | goAway; input audio is raw little-endian 16-bit PCM at 16kHz ("audio/pcm;rate=16000"), output audio is 24kHz PCM. Re-verify before trusting this code — the Live API is preview-tier and moves.
Index ¶
- Constants
- Variables
- func MaskKey(k string) string
- func RealtimeAudioMessage(b64 string) map[string]any
- func SanitizeSchema(raw json.RawMessage) (json.RawMessage, error)
- func ToolResponseMessage(responses []FunctionResponse) map[string]any
- type Content
- type Creator
- func (c *Creator) Create(_ context.Context, _ string) (voice.Session, error)
- func (c *Creator) DialURL() string
- func (c *Creator) Models() []string
- func (c *Creator) SetupMessage() map[string]any
- func (c *Creator) Verify(ctx context.Context) error
- func (c *Creator) WithModelSelection(sel string) *Creator
- func (c *Creator) WithSessionInstructions(text string) *Creator
- type FunctionCall
- type FunctionResponse
- type GenerationConfig
- type GoAway
- type InlineData
- type Option
- type Part
- type ServerContent
- type ServerMessage
- type Setup
- type Tool
- type ToolCall
- type ToolCancel
- type ToolDeclarations
- type Transcription
Constants ¶
const DefaultModel = "gemini-3.1-flash-live-preview"
DefaultModel is the Live API native-audio model. Override with GEMINI_LIVE_MODEL when the account has a newer one; Verify checks whatever is configured against the key, and the override deliberately bypasses LiveModels (see there).
It is itself a preview, and a dated preview being retired is the usual way this constant goes stale. That failure mode is survivable because GEMINI_LIVE_MODEL repoints the server at any Live model with no code change, and because Verify turns a retirement into a named boot error instead of a dead microphone.
const InputMimeType = "audio/pcm;rate=16000"
InputMimeType is the realtimeInput audio format the relay forwards.
const InputSampleRate = 16000
InputSampleRate is the rate the relay expects mic audio at, matching InputMimeType.
const LiveMethod = "bidiGenerateContent"
LiveMethod is the generation method a Live (BidiGenerateContent) session needs. Most Gemini models exist and answer GET /models/<name> while having no Live support at all, so "the model exists" is not the check worth making.
const LiveURL = "wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent"
LiveURL is the BidiGenerateContent WebSocket endpoint. The API key is passed as a query parameter on dial, the way the GenAI SDKs do it.
const OutputSampleRate = 24000
OutputSampleRate is the rate the Live API streams output audio at. The browser builds its playback AudioContext at this rate; it is here so the server and the page cannot drift apart silently.
const SessionTTL = 30 * time.Minute
SessionTTL bounds how long a created session descriptor stays usable before the browser must ask for a new one.
Variables ¶
var LiveModels = []string{ DefaultModel, }
LiveModels are the models this build offers the *browser* for per-session selection. It is an allowlist against a browser naming an arbitrary model, not a catalog of what works — GEMINI_LIVE_MODEL bypasses it entirely, because an operator setting an env var on their own server is a different trust level from a page choosing a model, and Verify still gates whatever they picked. Models keeps such an override selectable so switching away from it is not a one-way door.
Re-probe before adding to this list. Most models the key advertises do not serve bidiGenerateContent at all, and some that do are not assistants — a translate-preview model never emits a toolCall or a turnComplete, so offering it would fail at the microphone rather than at startup.
Functions ¶
func MaskKey ¶
MaskKey renders a key as head***tail plus its length — enough to tell two keys apart in a log (a stale shell export vs the one in the environment) without ever printing a usable credential. Short strings are shown as *** only.
func RealtimeAudioMessage ¶
RealtimeAudioMessage wraps one chunk of raw PCM16@16k mic audio (already base64-encoded) as a realtimeInput client message.
func SanitizeSchema ¶
func SanitizeSchema(raw json.RawMessage) (json.RawMessage, error)
SanitizeSchema rewrites a JSON Schema into the OpenAPI-3.0 subset that google.ai.generativelanguage.v1beta.Schema accepts. That message is parsed by a strict proto-JSON parser: any field it does not know closes the live session with WebSocket 1007 at setup, before a single microphone byte is sent.
Two constructs Go schema inference commonly emits are fatal there:
- "additionalProperties" — emitted for every struct; the field does not exist in Gemini's Schema at all. Dropped at every depth. Nothing is lost that Gemini could have enforced, and unknown arguments are still rejected server-side when the call is decoded.
- an array-valued "type" — emitted for every pointer or slice field ("type": ["null","array"]); Gemini's type is a single enum. Collapsed to its non-null member, with "nullable": true when "null" was present.
It is deliberately permissive and total: it strips only what it knows about and passes everything else through, including input that is not a JSON object at all. A keyword Gemini would reject but this function does not know must fail a guard test, not disable voice at startup.
It operates on marshaled bytes and returns fresh bytes, so it can never mutate a schema value shared with pi's other tool paths.
func ToolResponseMessage ¶
func ToolResponseMessage(responses []FunctionResponse) map[string]any
ToolResponseMessage answers one or more provider function calls.
Types ¶
type Creator ¶
type Creator struct {
APIKey string
Model string
Instructions string
// InstructionsFunc, when set, is consulted when the relay opens a session
// (SetupMessage) and wins over Instructions unless it returns empty. It is
// the hook for an instruction that depends on live server state.
InstructionsFunc func() string
Tools []Tool
ModelsURL string // override for tests; empty means the real endpoint
LiveURL string // override for tests; empty means the real endpoint
HTTPClient *http.Client
}
Creator holds the server-side Gemini Live configuration. It implements voice.SessionCreator, but does no provider round-trip per session: the relay dials the provider when the browser connects, so Create only shapes the transport descriptor the browser needs.
func (*Creator) Create ¶
Create implements voice.SessionCreator. The Realtime map tells the browser to use pi's relay transport; it carries no credential at all, because the relay authenticates by web session and dials the provider itself.
func (*Creator) DialURL ¶
DialURL is the provider WebSocket URL including the key. Only the server ever sees this string — never log it.
func (*Creator) Models ¶
Models is what this Creator offers the browser to choose from: the verified allowlist, plus the configured model when an operator pinned something outside it. An operator override that could not be selected back after switching away would be a trap.
func (*Creator) SetupMessage ¶
SetupMessage builds the first client message of a session.
func (*Creator) Verify ¶
Verify checks the key and the model against the models endpoint: the key is accepted, the model exists, and the model actually serves the Live API. An invalid key fails here with the provider's own words instead of surfacing later as a dial failure only the server log would see.
The Live check is the one that matters at the microphone. A model that exists but lacks bidiGenerateContent completes the HTTP verification and then kills every session mid-conversation with a WebSocket 1007 the browser has to explain — so verification asks the models endpoint what the model can do, not merely whether it is there.
func (*Creator) WithModelSelection ¶
WithModelSelection returns a shallow copy of c bound to one browser-selected model, or c itself when the selection is empty, unknown, or already current.
The copy is the point: two tabs must be able to run different models at once, so a per-session choice must never mutate the shared Creator. So is the allowlist check — SetupMessage's contract is that the browser cannot choose the model, and that survives here because the browser picks an entry from a server-defined list rather than supplying a model string. A stale tab naming a model this build no longer offers falls back to the default instead of failing the session.
func (*Creator) WithSessionInstructions ¶
WithSessionInstructions returns a copy of c whose system instruction is the given text, or c itself when the text is empty.
It is always a copy, never a mutation, for the same reason WithModelSelection is: two tabs run two live sessions off one shared Creator, and a per-session instruction — which is what an instruction carrying the current terminal's contents necessarily is — must not become every session's instruction. Empty is a deliberate no-op so a caller that has nothing session-specific to say falls through to whatever WithInstructions or InstructionsFunc configured.
type FunctionCall ¶
type FunctionCall struct {
ID string `json:"id,omitempty"`
Name string `json:"name"`
Args json.RawMessage `json:"args,omitempty"`
}
FunctionCall is one requested call.
type FunctionResponse ¶
type FunctionResponse struct {
ID string `json:"id,omitempty"`
Name string `json:"name"`
Response map[string]any `json:"response"`
}
FunctionResponse is one function result, matched to its call by ID.
type GenerationConfig ¶
type GenerationConfig struct {
ResponseModalities []string `json:"responseModalities,omitempty"`
}
GenerationConfig carries the response modalities the session asks for.
type GoAway ¶
type GoAway struct {
TimeLeft string `json:"timeLeft,omitempty"`
}
GoAway warns that the provider is about to end the session.
type InlineData ¶
InlineData is a base64 blob with its MIME type.
type Option ¶
type Option func(*Creator)
Option configures a Creator.
func WithHTTPClient ¶
WithHTTPClient overrides the HTTP client (tests only).
func WithInstructions ¶
WithInstructions sets the live-session system instructions.
func WithInstructionsFunc ¶
WithInstructionsFunc sets a per-session instructions source (see the field).
func WithLiveURL ¶
WithLiveURL overrides the Live WebSocket endpoint (tests only).
func WithModelsURL ¶
WithModelsURL overrides the models endpoint (tests only).
type Part ¶
type Part struct {
Text string `json:"text,omitempty"`
InlineData *InlineData `json:"inlineData,omitempty"`
}
Part is one content part: text or inline binary data.
type ServerContent ¶
type ServerContent struct {
ModelTurn *Content `json:"modelTurn,omitempty"`
Interrupted bool `json:"interrupted,omitempty"`
TurnComplete bool `json:"turnComplete,omitempty"`
InputTranscr *Transcription `json:"inputTranscription,omitempty"`
OutputTranscr *Transcription `json:"outputTranscription,omitempty"`
GenerationDone bool `json:"generationComplete,omitempty"`
}
ServerContent is the model's side of one turn.
func (*ServerContent) AudioParts ¶
func (sc *ServerContent) AudioParts() []string
AudioParts returns the base64 PCM payloads of a model turn, in order. The Live API streams output audio as inlineData parts at OutputSampleRate.
type ServerMessage ¶
type ServerMessage struct {
SetupComplete *struct{} `json:"setupComplete,omitempty"`
ServerContent *ServerContent `json:"serverContent,omitempty"`
ToolCall *ToolCall `json:"toolCall,omitempty"`
ToolCancel *ToolCancel `json:"toolCallCancellation,omitempty"`
GoAway *GoAway `json:"goAway,omitempty"`
}
ServerMessage is one BidiGenerateContentServerMessage. Exactly one of the pointer fields is set.
type Setup ¶
type Setup struct {
Model string `json:"model"`
GenerationConfig *GenerationConfig `json:"generationConfig,omitempty"`
SystemInstruction *Content `json:"systemInstruction,omitempty"`
Tools []ToolDeclarations `json:"tools,omitempty"`
InputTranscribe *struct{} `json:"inputAudioTranscription,omitempty"`
OutputTranscribe *struct{} `json:"outputAudioTranscription,omitempty"`
}
Setup is the BidiGenerateContentSetup message the relay sends first. The server owns it entirely — the browser cannot choose the model, instructions, or tools.
type Tool ¶
type Tool struct {
Name string `json:"name"`
Description string `json:"description"`
Parameters json.RawMessage `json:"parameters,omitempty"`
}
Tool mirrors the GenAI functionDeclarations shape for one exposed capability.
type ToolCall ¶
type ToolCall struct {
FunctionCalls []FunctionCall `json:"functionCalls,omitempty"`
}
ToolCall carries the function calls the model wants executed.
type ToolCancel ¶
type ToolCancel struct {
IDs []string `json:"ids,omitempty"`
}
ToolCancel names calls the model no longer wants answered.
type ToolDeclarations ¶
type ToolDeclarations struct {
FunctionDeclarations []Tool `json:"functionDeclarations,omitempty"`
}
ToolDeclarations wraps the function declarations for one tools entry.
type Transcription ¶
type Transcription struct {
Text string `json:"text,omitempty"`
}
Transcription is one transcript fragment. Gemini streams true increments for both directions, not whole utterances.