realtime

package
v0.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 16, 2026 License: MIT Imports: 9 Imported by: 0

Documentation

Overview

Package realtime speaks the subset of the OpenAI-Realtime websocket protocol an external voice backend needs: a session handshake, inbound audio append, and the server events that carry audio, speech boundaries, and transcripts.

Audio crosses this package as a **base64 string, never decoded**. The carrier delivers base64 G.711 mu-law and the protocol's audio fields carry base64 audio, so the two are the same bytes; decoding to re-encode would spend CPU per 20 ms frame to reproduce the input. Nothing here imports a G.711 codec.

Index

Constants

View Source
const (
	EventSessionUpdate = "session.update"
	EventAudioAppend   = "input_audio_buffer.append"
)

Client event types (this package -> backend).

View Source
const (
	EventSessionCreated  = "session.created"
	EventSpeechStarted   = "input_audio_buffer.speech_started"
	EventSpeechStopped   = "input_audio_buffer.speech_stopped"
	EventTranscriptDelta = "conversation.item.input_audio_transcription.delta"
	EventTranscriptDone  = "conversation.item.input_audio_transcription.completed"
	EventAudioDelta      = "response.output_audio.delta"
	EventResponseDone    = "response.done"

	// EventError is how a backend refuses something. This package acts on it
	// nowhere — it is named only so Dial can PREFER such a frame when saying
	// why a handshake never completed (AATK-136). Deliberately not a filter:
	// a backend refusing an engine extension is under no obligation to label
	// its refusal, and filtering would leave exactly those callers with a
	// bare deadline and nothing to go on.
	EventError = "error"
)

Server event types (backend -> this package). Any type not listed here is ignored by the read loop rather than treated as an error: the protocol is larger than the subset this package needs, and a backend is free to emit more of it.

View Source
const FormatG711ULaw = "audio/pcmu"

FormatG711ULaw is the session audio format for 8 kHz G.711 mu-law. The format identifier carries no sample rate because the codec fixes it.

View Source
const ItemTypeFunctionCall = "function_call"

ItemTypeFunctionCall is the "type" of a response output item that is a tool call rather than speech. It is not an event type — it appears INSIDE a response.done's output list — and it is named here beside the event types because it is read for the same reason they are: to tell whether a turn has actually ended.

View Source
const XClientSessionID = "X-Client-Session-Id"

XClientSessionID is the HTTP header under which the speech-to-speech fork stamps the client_session_id given to WithSessionID on every outbound chat-completions request, so a proxy in front of the model can tie each request to its session. The fork defines the same string as CLIENT_SESSION_HEADER; Go cannot import it, so the copy here is pinned by a test and a rename must be made on both sides together.

Variables

This section is empty.

Functions

func BuildVoiceUpdate

func BuildVoiceUpdate(voice string) ([]byte, error)

BuildVoiceUpdate marshals the minimal session.update that changes the backend's OUTPUT voice mid-call and touches nothing else:

{"type":"session.update","session":{"type":"realtime","audio":{"output":{"voice":"<id>"}}}}

It is deliberately NOT a parameterisation of newSessionUpdate. That constructor builds the dial handshake, whose job is to declare the audio format on both directions; its audioChannel.Format has no omitempty because the handshake always sets it. Reusing it here would send two empty format types, and a backend that merges only the fields it was sent would take them literally — clearing the negotiated format on a call already carrying audio. Hence a separate, minimal shape.

Exported, unlike buildSessionUpdate beside it, because its caller is in another package: the handshake is built and sent inside Dial, but the mid-call frame is built for HandleStreamRealtime to write through Client.Send. Send itself stays a raw client method — this adds a shape the engine can hand it, it does not restrict what else may be sent.

The voice reaches the wire exactly as supplied: not validated, trimmed, or case-folded. Which names are legal is the backend's to say, the same rule WithVoice's doc comment states for the dial voice.

func LogDrop

func LogDrop(n int) bool

LogDrop reports whether the nth drop of a call should be logged: the first one always, then every dropLogEvery-th. Exported so the twilio drain applies one policy rather than a second copy of it (universal §5).

func ResponseEndedInFunctionCall

func ResponseEndedInFunctionCall(raw json.RawMessage) bool

ResponseEndedInFunctionCall reports whether a response.done frame's output items include a function call — the case where the turn is NOT over and the caller's wait continues into the tool round trip plus whatever the model says afterwards.

"Include", not "end with": a response carrying speech AND a function call answers true. The question is whether anything more is coming, and a tool call means it is, whatever else the model said on the way.

Exported so that more than one thing can ask it and get the same answer. Inside this module the relay's filler audio reads it to decide whether to keep a hold loop running. It is exported for a consumer that has to place the same boundary from the outside — one recording what the relay did, say. A second implementation of this shape would drift silently: both answers look plausible in isolation, and a disagreement surfaces only as one wait reported as two, or a loop stopped in the middle of one.

It lives here rather than in the transport package that first needed it because it is protocol knowledge, not transport knowledge: it reads the realtime response object, and ItemTypeFunctionCall is declared above.

IT CHECKS THE FRAME'S OWN TYPE, so it is safe to hand every server event rather than only the ones already known to be a response.done. Without that check the exported contract would carry an unstated precondition, and the caller most likely to miss it is exactly the one this is exported for: a consumer piping every frame through it would get true for anything that happened to nest response.output[].type == "function_call".

Read from the raw frame rather than from a modelled field because this package models only the subset of the protocol it acts on, and one bool about one event type is not a reason to grow a decoded shape for the response object. The depth is load-bearing and not incidental: the item type is read at response.output[].type and nowhere else, so a frame merely containing the token — inside a transcript, or at the wrong nesting — answers false.

A frame that does not parse, or that carries no output items, reports false. The conservative answer is "the turn ended", which leaves a filler loop stopped rather than playing over whatever comes next — a caller hearing silence a moment early is a smaller fault than one hearing music over the reply.

Types

type Bridge

type Bridge struct {
	// contains filtered or unexported fields
}

Bridge pumps between a carrier and a Client. It performs no audio conversion in either direction — see the package comment.

One goroutine may call Forward and another may call Run concurrently; that is the intended shape, and it is safe because the underlying connection permits one concurrent reader and one concurrent writer. Concurrent calls to Forward are NOT safe — serialise them in the caller, which owns the carrier's frame ordering anyway.

func NewBridge

func NewBridge(c *Client, sink MediaSink) *Bridge

NewBridge wires a client to a carrier media sink.

func (*Bridge) Activity

func (b *Bridge) Activity() <-chan struct{}

Activity signals once per successful backend read, before that event is dispatched to any sink or the transcript channel — so it covers every event type Run observes, including ones dispatch's switch does not model and drops into default:. A caller wanting a liveness bound on the backend (as opposed to specific event types) should watch this rather than the sink or Transcripts, both of which only see a subset.

The channel is buffered by one and every send is non-blocking: a caller that misses a tick sees the next one, and Run's read loop can never stall waiting on a consumer that is not currently receiving.

func (*Bridge) Events

func (b *Bridge) Events() <-chan ServerEvent

Events yields every server event Run reads, including ones dispatch's switch does not model (see the default: branch in dispatch above) — the same coverage Activity documents, carried as data rather than a signal. Events are published in arrival order, before the event reaches dispatch's switch, so a modelled event that is not dropped reaches both Events() and its existing destination. Dropping is one-sided: it costs the event its place here and never its place in dispatch's switch, so a dropped transcript event still reaches Transcripts and a dropped audio delta still reaches the MediaSink.

Delivery never blocks Run. A caller that stops draining loses events rather than stalling the call: once the 16-event buffer is full the publish drops. Every drop is counted, and the count is logged at a bounded rate — the first drop of a call and every dropLogEvery-th after it, each line carrying the running total, so no loss is invisible and none of it is logged per drop at audio rate. That is deliberate — every event is published, including one audio delta per 20 ms frame, and Run's read loop is also what drives the MediaSink, so a blocking publish would silence audio within ~320 ms of a consumer looking away. The channel is closed when Run returns.

func (*Bridge) Forward

func (b *Bridge) Forward(ctx context.Context, payload string) error

Forward sends one inbound carrier frame's base64 payload to the backend. One frame in, one append out: no batching, no reordering, no rebuffering.

func (*Bridge) Run

func (b *Bridge) Run(ctx context.Context) error

Run reads server events until the connection ends or ctx is cancelled, driving the sink: audio deltas become Media, speech starts become Clear, and transcripts are published on Transcripts. Every event, of every type, is also offered to Events, whose delivery never blocks this loop: a caller that stops draining loses events rather than stalling the call; see Events. It always returns a non-nil error describing why it stopped — a backend that goes away is a fact the caller must see, never a silent return.

A second concurrent Run is refused rather than allowed to proceed: two readers on one connection is undefined, and the loser would panic closing an already-closed channel on the way out — both Transcripts and Events are closed here, so there are now two ways for that to go wrong and one guard covering both.

func (*Bridge) Transcripts

func (b *Bridge) Transcripts() <-chan Transcript

Transcripts yields transcription results in arrival order. The channel is closed when Run returns.

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client is one websocket connection to a realtime voice backend. It owns the connection's lifetime only in the sense that Close shuts it down; it never reconnects. Reconnect policy belongs to whatever owns the call, which knows whether a dropped backend should end the call or be retried.

func Dial

func Dial(ctx context.Context, url string, opts ...DialOption) (*Client, error)

Dial connects to url, sends the session.update handshake, and waits for the backend's session.created before returning. A Client that comes back from Dial has a negotiated session; on any failure the connection is torn down and the returned Client is nil, so a half-open Client can never escape and defer its error to the first frame of a live call.

func (*Client) AppendAudio

func (c *Client) AppendAudio(ctx context.Context, payload string) error

AppendAudio forwards one carrier frame's payload to the backend. payload is base64 G.711 and is placed on the wire unchanged.

func (*Client) Close

func (c *Client) Close() error

Close shuts the connection down. It is safe on a nil Client and on one whose connection has already gone away.

func (*Client) Read

func (c *Client) Read(ctx context.Context) (ServerEvent, error)

Read returns the next server event. Events whose type this package does not model decode with their Type set and the rest zero — Read never fails on an unrecognised type, because the protocol is larger than the subset used here and a backend may legitimately emit more of it. It returns an error when the connection ends.

func (*Client) Send

func (c *Client) Send(ctx context.Context, event json.RawMessage) error

Send writes one consumer-supplied client event to the backend, verbatim. This package does not model, validate, or interpret event — the consumer owns its own protocol correctness (session.update, conversation.item.create, response.create, function-call output, or anything else the backend accepts).

event's bytes go on the wire exactly as supplied: unlike send(), this does NOT go through json.Marshal, which would re-escape '<', '>', '&' and recompact whitespace on a json.RawMessage. It goes through the same write chokepoint as AppendAudio, so the two are safe to call concurrently.

type DialOption

type DialOption func(*dialConfig)

DialOption configures the session negotiated by Dial. Supplying none produces the handshake this package sent before options existed.

func WithHTTPClient

func WithHTTPClient(hc *http.Client) DialOption

WithHTTPClient overrides the HTTP client used for the WebSocket dial's handshake, giving control over the underlying transport.

This option exists FOR TESTING ONLY — specifically, to inject a transport-level fault (e.g. a connection whose Write calls can be made to fail on demand while Read keeps working) so a test can drive send-failure handling deterministically. Nil (the default) uses coder/websocket's own default transport. Production callers should never need this option; if you find yourself reaching for it outside a test, that is a sign this package needs a different seam.

func WithInstructions

func WithInstructions(s string) DialOption

WithInstructions sets the session persona the backend is told to adopt. The protocol's session object defines this field; a backend that builds its runtime config from the session request reads it and prepends it as a system message. Empty omits the field rather than sending "".

func WithSessionID

func WithSessionID(id string) DialOption

WithSessionID puts the consumer's OWN identifier for this session into the handshake (AATK-136), under the wire name sessionSpec.ClientSessionID's struct tag states. A backend that forwards this session's work onward — to a proxy, a context service, anything holding per-session state — has nothing in the request identifying which session it belongs to; this is that identifier, and what the backend does with it is entirely the backend's and the consumer's business.

The value is opaque to this package: it is neither generated, validated, nor normalized here, and it is never read back off the wire. Empty (the default) omits the field rather than sending "".

A plain string is the right shape HERE and needs no resolver twin: this package has no entry point that binds options once and replays them across sessions, so a caller is always able to supply the right value at dial time — one Dial, one session, one identifier. Able, not forced; hoisting an option slice to a package variable would still share one identifier across dials. That is a property of this layer rather than a general one, and the layer that does bind options once has twilio.WithSessionIDFor for it.

func WithTools

func WithTools(tools json.RawMessage) DialOption

WithTools declares the consumer's tool definitions for this session (AATK-85). This package models nothing about what a tool is — the consumer owns tool schemas entirely — so tools is carried as json.RawMessage rather than a typed struct: a typed struct would re-encode the consumer's bytes on the way to the wire (reordering keys, dropping unknown fields, re-escaping '<', '>', '&'), and the declared tools must reach the backend unmodified. Nil (the default) omits the field entirely, producing the handshake this package sent before tools existed. A tools value that is not syntactically valid JSON makes Dial return an error rather than sending a malformed handshake.

func WithVoice

func WithVoice(name string) DialOption

WithVoice sets the backend's output voice for the session. The protocol's session object defines this field on the output side of the audio spec; a backend that supports multiple voices reads it and answers in the named one. This package does not validate or normalize the name — the backend owns which names it accepts, so whatever is supplied here reaches the wire unmodified. Empty omits the field rather than sending "".

type MediaSink

type MediaSink interface {
	// Media delivers one chunk of outbound audio, base64-encoded.
	Media(ctx context.Context, payload string) error
	// Clear discards audio already buffered toward the caller. It is the
	// barge-in signal.
	Clear(ctx context.Context) error
}

MediaSink is the carrier-facing side of the bridge: where synthesized audio and barge-in signals go. Payloads are base64 G.711, forwarded unchanged.

type ServerEvent

type ServerEvent struct {
	Type       string          `json:"type"`
	Delta      string          `json:"delta,omitempty"`
	Transcript string          `json:"transcript,omitempty"`
	Raw        json.RawMessage `json:"-"`
}

ServerEvent is one decoded server event, flattened to the fields this package reads. Delta carries base64 audio for EventAudioDelta; Transcript carries text for the transcription events.

Raw carries the whole frame exactly as it arrived on the wire, for every event type — including ones this package does not model into Type/Delta/ Transcript. It has no json tag mapping any wire field onto it: nothing in the protocol names "the whole document", so Client.Read must assign it explicitly after decoding, from the same bytes it unmarshalled.

type Transcript

type Transcript struct {
	Text  string
	Final bool
}

Transcript is one transcription result for caller speech. Final marks the completed transcript for an utterance; partials arrive with Final false.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL