aatoolkit

package module
v0.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 15, 2026 License: MIT Imports: 0 Imported by: 0

README

aatoolkit

An open Go engine for building voice and telephony AI agents.

aatoolkit is the mechanism of a phone agent: it answers the call, moves the audio, decides when the caller has finished speaking, drives the LLM, and keeps track of what it learns. A specific agent supplies the meaning — prompts, persona, policy, tools, ontology — from its own repository. Any number of agents can run on the same engine.

go get github.com/iansmith/aatoolkit

Key features

Area Package What it gives you
Telephony telephony/twilio Twilio voice webhooks, signature validation, Media Streams over WebSocket, inbound SMS and outbound SMS via the REST API.
Speech-to-speech relay telephony/realtime, telephony/twilio Bridges a phone call to any OpenAI-Realtime-compatible voice backend, with no transcoding. See below.
Speech pipeline telephony Silero VAD, an STT router, and a turn-taking state machine that decides when the caller is done — with tunable silence/utterance/turn caps.
LLM driver driver, host OpenAI-compatible LLM client with tiered models and streaming, TTS synthesis and a serial speech queue, health and Twilio HTTP servers.
Hot-reloadable policy interp Loads agent policy written in plain Go at runtime (via yaegi) against a host interface you choose. Change behavior without rebuilding or redeploying the engine.
Fact extraction extract Wire contract for an entity-extraction sidecar (e.g. GLiNER), for turning conversation into stored facts.
Process supervisor cmd/aa-server-status Launches, health-checks, and tears down the engine and its model sidecars from one TOML config; interactive REPL or --auto up|down for systemd.
Test tooling telephony replay, cmd/probeset, cmd/twilio-cli Replay recorded calls through the turn-taking logic, record per-decision event logs, and drive a local server as if it were Twilio — voice or SMS, no account or phone needed.

How an agent plugs in

The engine never imports the agent. The agent injects itself through three seams:

  1. driver.Config — model tiers, TTS transport, and a system-prompt provider.
  2. interp.Load — the directory of Go policy source to interpret, plus the symbols it may import.
  3. host.Host — the interface the interpreted policy calls back into (LLM transport by tier).

Dependencies flow one way: agent → aatoolkit, never the reverse.

Speech-to-speech

Besides its own VAD → STT → LLM → TTS pipeline, aatoolkit can hand a call to an external speech-to-speech backend that speaks the OpenAI Realtime WebSocket protocol — for example the open-source speech-to-speech server (a fork of huggingface/speech-to-speech), which runs a fully local, swappable VAD → STT → LLM → TTS stack behind that protocol.

In this mode the engine is a thin, careful relay between the phone carrier and the backend:

  • One switch. twilio.NewStreamHandler(realtimeURL, opts...) routes calls to the backend; an empty URL keeps the built-in pipeline. Consumers that own their handler can call twilio.HandleStreamRealtime directly.
  • No transcoding. Twilio's base64 G.711 μ-law goes to the backend as-is, and the reply audio comes back the same way. Barge-in is forwarded as a carrier clear.
  • Per-call identity. WithInstructions[For] sets the persona, WithVoice the voice, WithTools the tool schemas (passed through byte-for-byte), and WithSessionID[For] a session id the speech-to-speech server forwards on each LLM request as X-Client-Session-Id, so a proxy can tie requests to the call.
  • Observe everything. Channels for transcripts (WithTranscriptChan), every server event (WithServerEventChan), inbound caller audio, and every payload sent to the carrier.
  • Steer mid-call. Send your own client events (e.g. tool results) with WithClientEventChan, change the voice with WithVoiceUpdateChan, and use WithMarkRequestChan / WithMarkEchoChan to learn when the caller has actually heard a reply.
  • Call hygiene. WithFillerAudio plays a hold loop while the model is thinking or running a tool, WithIdleTimeout ends calls whose backend has gone quiet, and WithFarewellAudio plays a goodbye before hanging up.

Note that on this path the backend's VAD and turn-taking run the conversation; the engine's own turn-taking logic is bypassed for those calls.

handler := twilio.NewStreamHandler("ws://localhost:8765/v1/realtime",
    twilio.WithInstructions("You are a friendly receptionist."),
    twilio.WithIdleTimeout(30*time.Second),
)

Status

Early and moving. The public API will change until it settles. See design/ for architecture notes.

License

MIT — see LICENSE.

Documentation

Overview

Package aatoolkit is an open, reusable toolkit for building conversational AI assistants: telephony (Twilio media streams), speech-to-text and voice-activity detection, LLM transport, a dynamic-Go policy loader for hot-reloadable behavior, and a fact-database toolkit for extracting and storing knowledge in a graph.

It provides mechanism, not a finished assistant. The particular behavior, identity, and policy of any given assistant live in a separate, private repository that depends on this module — never the reverse.

Extraction of the reusable packages from their original monorepo is in progress; see design/ for the architecture and the split plan.

Directories

Path Synopsis
cmd
aa-server-status command
RealEngine is the reconciliation engine wiring launchers (internal/lifecycle), teardown (internal/lifecycle), observation (internal/observe), and health (internal/health) into the fleet verbs (design/aa-server-status.md §2, §3, §6.3, §6.5).
RealEngine is the reconciliation engine wiring launchers (internal/lifecycle), teardown (internal/lifecycle), observation (internal/observe), and health (internal/health) into the fleet verbs (design/aa-server-status.md §2, §3, §6.3, §6.5).
probeset command
twilio-cli command
twprobe command
wsecho command
Command wsecho is a throwaway WebSocket echo server for local testing of twilio-cli.
Command wsecho is a throwaway WebSocket echo server for local testing of twilio-cli.
Package config loads and validates aa-server-status's TOML configuration: a committed base file deep-merged with a gitignored local overlay for secrets.
Package config loads and validates aa-server-status's TOML configuration: a committed base file deep-merged with a gitignored local overlay for secrets.
Package driver is the compiled engine an agent runs on: the LLM HTTP client (Host, a host.Host implementation), TTS synthesis + playback, the serial speech queue, and health/Twilio HTTP servers.
Package driver is the compiled engine an agent runs on: the LLM HTTP client (Host, a host.Host implementation), TTS synthesis + playback, the serial speech queue, and health/Twilio HTTP servers.
Package extract provides the wire contract and interface for an extraction sidecar.
Package extract provides the wire contract and interface for an extraction sidecar.
Package host defines the surface that an interpreted chat policy needs from its compiled driver.
Package host defines the surface that an interpreted chat policy needs from its compiled driver.
internal
health
Package health implements the mandatory health gate for aa-server-status servers (design/aa-server-status.md §6.1 stage 3, §7.1) in the two forms a server may declare it: a single GET request whose 2xx response is the authoritative "serving" signal, or a command whose exit status of 0 is.
Package health implements the mandatory health gate for aa-server-status servers (design/aa-server-status.md §6.1 stage 3, §7.1) in the two forms a server may declare it: a single GET request whose 2xx response is the authoritative "serving" signal, or a command whose exit status of 0 is.
lifecycle
Package lifecycle implements the common child-launch machinery shared by every server type — process-group isolation, per-server environment overlay, and log-file management — plus the two launchers whose command shapes need no further logic (mlx, exec).
Package lifecycle implements the common child-launch machinery shared by every server type — process-group isolation, per-server environment overlay, and log-file management — plus the two launchers whose command shapes need no further logic (mlx, exec).
observe
Package observe is aa-server-status's observation layer: it answers "what is actually running" without ever killing or launching anything.
Package observe is aa-server-status's observation layer: it answers "what is actually running" without ever killing or launching anything.
Package interp is the generic mechanism for loading Go "policy" source at runtime: read a directory of .go files, compile them as a named package under a fresh yaegi interpreter, inject a set of host symbols the policy may import, and evaluate an expression (typically a function symbol) that the caller type-asserts and calls.
Package interp is the generic mechanism for loading Go "policy" source at runtime: read a directory of .go files, compile them as a named package under a fresh yaegi interpreter, inject a set of host symbols the policy may import, and evaluate an expression (typically a function symbol) that the caller type-asserts and calls.
assets
Package assets embeds vendored binary model assets so the telephony package never needs a runtime file read to load them.
Package assets embeds vendored binary model assets so the telephony package never needs a runtime file read to load them.
realtime
Package realtime speaks the subset of the OpenAI-Realtime websocket protocol an external voice backend needs: a session handshake, inbound audio append, and the server events that carry audio, speech boundaries, and transcripts.
Package realtime speaks the subset of the OpenAI-Realtime websocket protocol an external voice backend needs: a session handshake, inbound audio append, and the server events that carry audio, speech boundaries, and transcripts.
twilio/smsfake
Package smsfake is the fake-Twilio SMS round trip: it builds and signs the inbound-SMS webhook a real Twilio account would post to a server, and it stands in for the Twilio Messages REST API so the server's outbound reply can be captured and asserted on without a real account.
Package smsfake is the fake-Twilio SMS round trip: it builds and signs the inbound-SMS webhook a real Twilio account would post to a server, and it stands in for the Twilio Messages REST API so the server's outbound reply can be captured and asserted on without a real account.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL