rtvi

type AboutClientData ¶ added in v0.1.0

type AboutClientData struct {
	Library         string `json:"library"`
	LibraryVersion  string `json:"library_version,omitempty"`
	Platform        string `json:"platform,omitempty"`
	PlatformVersion string `json:"platform_version,omitempty"`
	PlatformDetails any    `json:"platform_details,omitempty"`
}

AboutClientData describes the client an RTVI session is with: which client library it uses, on what platform, and whatever else it cares to say. It is what a client sends with its client-ready, and the same shape a bot-ready sends back about itself.

type AudioLevelData ¶ added in v0.1.0

type AudioLevelData struct {
	// Value is the volume on the 0..1 scale audio/loudness measures.
	Value float64 `json:"value"`
}

AudioLevelData is how loud one side of the conversation currently is.

type BotReadyData ¶

type BotReadyData struct {
	Version string           `json:"version"`
	About   *AboutClientData `json:"about,omitempty"`
}

BotReadyData is the payload of a bot-ready message: the protocol version the session settled on, and what the bot is.

type ClientMessageFrame ¶ added in v0.1.0

type ClientMessageFrame struct {
	frames.BaseSystemFrame
	// MsgID is the client's id for the request, echoed in the answer.
	MsgID string
	// Type is the client's own message type, opaque to the protocol.
	Type string
	// Data is whatever the client sent with it, left as raw JSON so the
	// processor answering it decodes the shape it expects.
	Data json.RawMessage
}

ClientMessageFrame carries a client-message: something the client asked the bot that the protocol has no message of its own for. It travels downstream from the RTVI processor, so a processor placed anywhere in the pipeline can answer it by pushing a ServerResponseFrame back.

A client message expects an answer. Answer it with a ServerResponseFrame naming this frame, so the client can pair the two.

func NewClientMessageFrame(msgID, msgType string, data json.RawMessage) *ClientMessageFrame

func (f *ClientMessageFrame) String() string

type ClientReadyData ¶ added in v0.1.0

type ClientReadyData struct {
	Version string          `json:"version"`
	About   AboutClientData `json:"about,omitzero"`
}

ClientReadyData is the payload of a client-ready message: the protocol version the client speaks, and what it is.

func ParseClientReadyData(raw json.RawMessage) (ClientReadyData, error)

type ConfigureObserverFrame ¶ added in v0.1.0

type ConfigureObserverFrame struct {
	frames.BaseSystemFrame
	// FunctionCallReportLevel is the per-function report-level map to apply, or
	// nil to leave the observer's current map unchanged.
	FunctionCallReportLevel map[string]FunctionCallReportLevel
	// VADUserSpeakingEnabled turns the raw VAD speaking events on or off, or is
	// nil to leave the observer's current setting unchanged.
	VADUserSpeakingEnabled *bool
}

ConfigureObserverFrame reconfigures a running Observer. It lets a trusted, server-side source adjust what the observer exposes at runtime without baking the setting into the bot, where it would apply to every client. Only the fields that are set are applied; a nil field leaves the current configuration unchanged.

The eval harness pushes this, through the eval-only serializer, to raise the function-call report level for the calls a scenario asserts on, so production bots can keep the secure default.

func NewConfigureObserverFrame(
	level map[string]FunctionCallReportLevel, vadUserSpeaking *bool,
) *ConfigureObserverFrame

func (f *ConfigureObserverFrame) String() string

type DTMFData ¶ added in v0.1.0

type DTMFData struct {
	Buttons []string `json:"buttons"`
}

DTMFData is the payload of a dtmf message: the keypad keys the client pressed, in the order they were pressed.

type ErrorData ¶

type ErrorData struct {
	Error string `json:"error"`
	Fatal bool   `json:"fatal"`
}

ErrorData is the payload of an error message.

type ErrorResponseData ¶ added in v0.1.0

type ErrorResponseData struct {
	Error string `json:"error"`
}

ErrorResponseData is the payload of an error-response.

type FunctionCallReportLevel ¶ added in v0.1.0

type FunctionCallReportLevel string

FunctionCallReportLevel is how much of a tool call is exposed in the RTVI function-call events. A call's name and its arguments can carry information a client has no business seeing, so what is reported is a per-function setting rather than a fixed payload.

const (
	// ReportDisabled emits no function-call event at all for the call.
	ReportDisabled FunctionCallReportLevel = "disabled"
	// ReportNone emits the event with the tool call id only. This is the default:
	// a client learns that a call is running, and nothing more.
	ReportNone FunctionCallReportLevel = "none"
	// ReportName adds the function's name, still without arguments or result.
	ReportName FunctionCallReportLevel = "name"
	// ReportFull adds the function's name, its arguments and its result.
	ReportFull FunctionCallReportLevel = "full"
)

type FunctionCallResultData ¶ added in v0.1.0

type FunctionCallResultData struct {
	FunctionName string          `json:"function_name"`
	ToolCallID   string          `json:"tool_call_id"`
	Arguments    json.RawMessage `json:"arguments"`
	Result       json.RawMessage `json:"result"`
}

FunctionCallResultData is the payload of an llm-function-call-result: a tool call the client ran on the bot's behalf, and what it produced.

func ParseFunctionCallResultData(raw json.RawMessage) (FunctionCallResultData, error)

func (d FunctionCallResultData) ResultText() string

type Incoming ¶

type Incoming struct {
	Label string          `json:"label"`
	Type  string          `json:"type"`
	ID    string          `json:"id"`
	Data  json.RawMessage `json:"data"`
}

Incoming is a received RTVI message with its data left as raw JSON for type-specific decoding.

func ParseIncoming(raw []byte) (Incoming, error)

type LLMFunctionCallData ¶ added in v0.1.0

type LLMFunctionCallData struct {
	ToolCallID   string          `json:"tool_call_id"`
	FunctionName string          `json:"function_name,omitempty"`
	Arguments    json.RawMessage `json:"arguments,omitempty"`
}

LLMFunctionCallData is the payload of a llm-function-call-in-progress message. The tool call id is always present; the name and the arguments are omitted unless the observer's report level for the function allows them (see FunctionCallReportLevel), because either can carry information a client has no business seeing.

type LLMFunctionCallStartData ¶ added in v0.1.0

type LLMFunctionCallStartData struct {
	FunctionName string `json:"function_name,omitempty"`
}

LLMFunctionCallStartData is the payload of a llm-function-call-started message: the model has asked for a call, before it begins executing. The name is omitted unless the observer's report level for the function allows it.

type LLMFunctionCallStoppedData ¶ added in v0.1.0

type LLMFunctionCallStoppedData struct {
	ToolCallID string `json:"tool_call_id"`
	// Canceled reports whether the call was canceled rather than completing. The
	// wire name keeps the protocol's spelling, which the clients already send.
	Canceled     bool   `json:"cancelled"` //nolint:misspell // the protocol spells it this way
	FunctionName string `json:"function_name,omitempty"`
	Result       string `json:"result,omitempty"`
}

LLMFunctionCallStoppedData is the payload of a llm-function-call-stopped message, sent when a call completes with a result or is canceled. As with the in-progress payload, the name and the result are omitted unless the observer's report level for the function allows them.

type Message ¶

type Message struct {
	Label string `json:"label"`
	Type  string `json:"type"`
	ID    string `json:"id,omitempty"`
	Data  any    `json:"data,omitempty"`
}

Message is the RTVI message envelope. Outgoing event messages omit id; bot-ready and responses echo the request id.

func BotAudioLevel(level float64) Message

func BotLLMText(text string) Message

func BotReady(id, version string, about *AboutClientData) Message

func BotTTSText(text string) Message

func BotTranscription(text string) Message

func Error(msg string, fatal bool) Message

func ErrorResponse(id, err string) Message

func LLMFunctionCall(name, toolCallID string, args json.RawMessage, level FunctionCallReportLevel) Message

func LLMFunctionCallStart(name string, level FunctionCallReportLevel) Message

func LLMFunctionCallStopped(
	name, toolCallID, result string, canceled bool, level FunctionCallReportLevel,
) Message

func Metrics(data MetricsData) Message

func ServerMessage(data any) Message

func ServerResponse(id, msgType string, data any) Message

func UserAudioLevel(level float64) Message

func UserTranscription(text, userID, timestamp string, final bool) Message

type MetricData ¶

type MetricData struct {
	Processor string  `json:"processor"`
	Value     float64 `json:"value"`
	Model     string  `json:"model,omitempty"`
}

MetricData is one timing or count entry in a metrics message (ttfb, processing or characters). Value is in seconds for timings, or a count.

type MetricsData ¶

type MetricsData struct {
	TTFB            []MetricData      `json:"ttfb,omitempty"`
	TTFA            []TTFAMetricData  `json:"ttfa,omitempty"`
	Processing      []MetricData      `json:"processing,omitempty"`
	Characters      []MetricData      `json:"characters,omitempty"`
	STTUsage        []MetricData      `json:"stt_usage,omitempty"`
	TextAggregation []MetricData      `json:"text_aggregation,omitempty"`
	Tokens          []TokenMetricData `json:"tokens,omitempty"`
	Turn            []TurnMetricData  `json:"turn,omitempty"`
}

MetricsData is the payload of a metrics message: each kind is a list so a single message can report several processors at once.

type Observer ¶ added in v0.1.0

type Observer struct {
	// contains filtered or unexported fields
}

Observer reports pipeline events to an RTVI client. It watches every frame handed between processors and turns the ones a client cares about into RTVI messages, which it hands to a Processor to send.

Only outgoing events come from here. Messages arriving from the client are handled by the Processor, which also carries out the handshake.

Watching rather than sitting in the pipeline is what lets a client be told about frames that never travel the whole chain, and lets an event be reported from where the frame actually is: text pushed by the LLM is seen at the LLM, whether or not anything downstream forwards it.

func NewObserver(sink *Processor) *Observer

func NewObserverWithParams(sink *Processor, params ObserverParams) *Observer

func (o *Observer) AddIgnoredSource(source processor.Processor)

func (o *Observer) OnPushFrame(data processor.FramePushed)

func (o *Observer) RemoveIgnoredSource(source processor.Processor)

type ObserverParams ¶ added in v0.1.0

type ObserverParams struct {
	// FunctionCallReportLevel maps a function name to the level of detail its
	// events carry. The "*" key sets the level for functions not listed; when it
	// is absent too, ReportNone applies.
	FunctionCallReportLevel map[string]FunctionCallReportLevel
	// BotLLMEnabled reports what the model produced: the brackets around a
	// response and the text inside it. Nil leaves it on.
	BotLLMEnabled *bool
	// BotTTSEnabled reports what the voice is doing: the brackets around a
	// synthesis and the text as it is spoken. Nil leaves it on.
	BotTTSEnabled *bool
	// BotSpeakingEnabled reports when the bot's audio starts and stops playing,
	// and when it was cut off. Nil leaves it on.
	BotSpeakingEnabled *bool
	// UserSpeakingEnabled reports the user's turn starting and ending. This is
	// the turn as the pipeline settled it, which a strategy may gate or defer;
	// VADUserSpeakingEnabled is the raw signal behind it. Nil leaves it on.
	UserSpeakingEnabled *bool
	// UserTranscriptionEnabled reports what the user said, both the interim
	// guesses and the final transcript. Nil leaves it on.
	UserTranscriptionEnabled *bool
	// MetricsEnabled reports the timings and usage the pipeline measures. Nil
	// leaves it on.
	MetricsEnabled *bool
	// IgnoredSources are processors whose frames the observer says nothing
	// about. It keeps a secondary branch of the pipeline out of the client's
	// view: an evaluation model answering alongside the real one has a whole
	// conversation of its own, and none of it is for the client to see. Sources
	// can be added and removed while the pipeline runs with AddIgnoredSource and
	// RemoveIgnoredSource.
	IgnoredSources []processor.Processor
	// VADUserSpeakingEnabled reports the raw VAD speaking signal as well as the
	// turn-level one. The two differ whenever a turn strategy gates or defers a
	// turn, which is what makes the raw signal useful as a timing anchor. Off by
	// default, because a client wants turns rather than the signal behind them.
	VADUserSpeakingEnabled bool
	// UserAudioLevelEnabled reports how loud the user is, for a client drawing a
	// speaking meter. Off by default: it is a message every AudioLevelPeriod for
	// as long as the call lasts, which a client that draws nothing does not want.
	UserAudioLevelEnabled bool
	// BotAudioLevelEnabled reports how loud the bot is. Off by default, for the
	// same reason.
	BotAudioLevelEnabled bool
	// AudioLevelPeriod is how often an audio level is reported while it is
	// enabled; zero defaults to 150 ms. Audio arrives far more often than a
	// meter can usefully be redrawn, and measuring loudness is not free, so the
	// level is reported on a period rather than per frame.
	AudioLevelPeriod time.Duration
}

ObserverParams configures what an Observer reports.

The categories a client normally wants are on unless they are turned off, so their fields are pointers: nil means on, and only an explicit false suppresses them. The ones that would flood a client that has not asked for them are off unless turned on, and are plain bools.

func DefaultObserverParams() ObserverParams

type Processor ¶

type Processor struct {
	*processor.Base
	// contains filtered or unexported fields
}

Processor bridges a pipeline to an RTVI client. It completes the handshake, replying to client-ready with bot-ready, and carries out what the client asks of the pipeline.

Place it at the top of the pipeline, ahead of the input transport. What the client injects is pushed downstream from there, so it travels the pipeline by the same path a real caller's input takes: a typed message reaches the context aggregator, a keypress reaches the DTMF handling, and each arrives in order with everything the turn is made of. Its own messages to the client are pushed downstream too, and reach the output transport at the far end.

It does not report pipeline events: pair it with an Observer, which watches the whole pipeline and sends through this processor. Incoming client messages arrive as InputTransportMessageFrames, which the input transport broadcasts so they reach this processor from either side; outgoing messages are pushed downstream as OutputTransportMessageUrgentFrames.

func NewProcessor() *Processor

func (p *Processor) Cleanup(ctx context.Context) error

func (p *Processor) ClientVersion() [3]int

func (p *Processor) ProcessFrame(ctx context.Context, f frames.Frame, dir processor.Direction) error

func (p *Processor) Send(msg Message)

func (p *Processor) SendErrorResponse(ctx context.Context, msg *ClientMessageFrame, reason string) error

func (p *Processor) SendServerMessage(ctx context.Context, data any) error

func (p *Processor) SendServerResponse(ctx context.Context, msg *ClientMessageFrame, data any) error

func (p *Processor) Setup(ctx context.Context, s processor.Setup) error

type RawAudioData ¶ added in v0.1.0

type RawAudioData struct {
	//nolint:tagliatelle // RTVI wire fields, camelCase in the protocol
	Base64Audio string `json:"base64Audio,omitempty"`
	//nolint:tagliatelle // RTVI wire fields, camelCase in the protocol
	Base64AudioBatch []string `json:"base64AudioBatch,omitempty"`
	//nolint:tagliatelle // RTVI wire fields, camelCase in the protocol
	SampleRate int `json:"sampleRate"`
	//nolint:tagliatelle // RTVI wire fields, camelCase in the protocol
	NumChannels int `json:"numChannels"`
}

RawAudioData is the payload of a raw-audio or raw-audio-batch message: audio the client captured itself, base64-encoded 16-bit PCM, one chunk or a batch of them. It is how a client that does its own capture feeds the pipeline over the message channel rather than a media track.

func ParseRawAudioData(raw json.RawMessage) (RawAudioData, error)

func (d RawAudioData) Chunks() []string

type RawClientMessageData ¶ added in v0.1.0

type RawClientMessageData struct {
	T string          `json:"t"`
	D json.RawMessage `json:"d,omitempty"`
}

RawClientMessageData is the payload of a client-message: the client's own message type and whatever it carries, both opaque to the protocol. It is how a client asks the bot something the protocol has no message for.

func ParseRawClientMessageData(raw json.RawMessage) (RawClientMessageData, error)

type RawServerResponseData ¶ added in v0.1.0

type RawServerResponseData struct {
	T string `json:"t"`
	D any    `json:"d,omitempty"`
}

RawServerResponseData is the payload of a server-response: the type of the client message being answered, so the client can pair the answer with what it asked, and the answer itself.

type SendTextData ¶ added in v0.1.0

type SendTextData struct {
	Content string           `json:"content"`
	Options *SendTextOptions `json:"options,omitempty"`
}

SendTextData is the payload of a send-text message: user text to inject into the conversation, with options controlling whether the LLM runs immediately.

func ParseSendTextData(raw json.RawMessage) (SendTextData, error)

func (d SendTextData) AudioResponse() bool

func (d SendTextData) RunImmediately() bool

type SendTextOptions ¶ added in v0.1.0

type SendTextOptions struct {
	RunImmediately *bool `json:"run_immediately,omitempty"`
	AudioResponse  *bool `json:"audio_response,omitempty"`
}

SendTextOptions controls how the pipeline processes a send-text message. Both fields default to true when absent, matching the RTVI client SDKs.

type ServerMessageFrame ¶ added in v0.1.0

type ServerMessageFrame struct {
	frames.BaseSystemFrame
	// Data is the message, serialized as the message's data.
	Data any
}

ServerMessageFrame carries an unprompted message to the client: whatever the bot wants to tell it that the protocol has no message for. Push one from anywhere in the pipeline and the Observer sends it.

func NewServerMessageFrame(data any) *ServerMessageFrame

func (f *ServerMessageFrame) String() string

type ServerResponseFrame ¶ added in v0.1.0

type ServerResponseFrame struct {
	frames.BaseSystemFrame
	// ClientMsg is the request being answered.
	ClientMsg *ClientMessageFrame
	// Data is the answer, serialized as the response's data.
	Data any
	// Error refuses the request, and is the reason given to the client. When it
	// is set, Data is not sent.
	Error string
}

ServerResponseFrame answers one ClientMessageFrame. It names the frame it answers, so the client can pair the answer with the request it made.

Setting Error refuses the request instead: the client receives an error-response rather than a server-response, so a request it made never simply goes unanswered.

func NewServerErrorResponseFrame(msg *ClientMessageFrame, reason string) *ServerResponseFrame

func NewServerResponseFrame(msg *ClientMessageFrame, data any) *ServerResponseFrame

func (f *ServerResponseFrame) String() string

type TTFAMetricData ¶ added in v0.1.0

type TTFAMetricData struct {
	Processor      string  `json:"processor"`
	Model          string  `json:"model,omitempty"`
	TTFA           float64 `json:"ttfa"`
	TTFB           float64 `json:"ttfb"`
	LeadingSilence float64 `json:"leading_silence"`
}

TTFAMetricData is one time-to-first-audible-sample entry, reported with the breakdown that makes it up: the time to first byte it builds on, and the silence padded on before the first audible sample. TTFB here is the same measurement reported under "ttfb", not another one.

type TextData ¶

type TextData struct {
	Text string `json:"text"`
}

TextData is the payload of text messages (bot-transcription, bot-tts-text, bot-llm-text).

type TokenMetricData ¶

type TokenMetricData struct {
	Processor        string `json:"processor"`
	Model            string `json:"model,omitempty"`
	PromptTokens     int64  `json:"prompt_tokens"`
	CompletionTokens int64  `json:"completion_tokens"`
	TotalTokens      int64  `json:"total_tokens"`
}

TokenMetricData is one LLM token-usage entry in a metrics message.

type TurnMetricData ¶ added in v0.1.0

type TurnMetricData struct {
	Processor    string  `json:"processor"`
	Complete     bool    `json:"complete"`
	Probability  float64 `json:"probability"`
	ProcessingMs float64 `json:"processing_ms"`
}

TurnMetricData is one end-of-turn prediction: whether the analyzer judged the turn finished, how confident it was, and how long deciding took.

type UserTranscriptionData ¶

type UserTranscriptionData struct {
	Text      string `json:"text"`
	UserID    string `json:"user_id"`
	Timestamp string `json:"timestamp"`
	Final     bool   `json:"final"`
}

UserTranscriptionData is the payload of a user-transcription message.