glyphoxa

module

v0.1.2 Latest Latest Go to latest Published: Mar 19, 2026 License: GPL-3.0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/MrWong99/glyphoxa

Links

Open Source Insights

README ¶

🐉 Glyphoxa

AI-Powered Voice NPCs for Tabletop RPGs — a platform-agnostic, provider-independent voice AI framework that brings your NPCs to life.

What is Glyphoxa?

Glyphoxa is a real-time voice AI framework that brings AI-driven talking personas into live voice chat sessions. Built for tabletop RPGs, it serves as a persistent AI co-pilot for the Dungeon Master — voicing NPCs with distinct personalities, transcribing sessions, and answering rules questions — without ever replacing the human storyteller.

Written in Go for native concurrency and sub-2-second mouth-to-ear latency.

⚠️ Early Alpha — Glyphoxa is under active development. APIs may change between commits.

✨ Features

🗣️ Voice NPC Personas — AI-controlled NPCs with distinct voices, personalities, and backstories that speak in real-time
🧠 Hybrid Memory System — NPCs remember. Hot layer for instant context, cold layer for deep history, knowledge graph for world state
🔧 MCP Tool Integration — Plug-and-play tools (dice, rules lookup, image gen, web search) with performance-budgeted execution
🔄 Provider-Agnostic — Swap LLM, STT, TTS, or audio platform with a config change, not a rewrite
⚡ Sub-2s Latency — End-to-end streaming pipeline with speculative pre-fetch and sentence-level TTS
🎭 Multi-NPC Orchestration — Multiple NPCs with address detection, turn-taking, and priority-based audio mixing
📜 Live Session Transcription — Continuous STT with speaker identification for session logging and future lookup
🧪 Dual-Model Sentence Cascade — Experimental: fast model opener + strong model continuation for perceived <600ms voice onset
🗺️ Entity Management — Pre-session world-building with YAML campaign files and VTT imports (Foundry VTT, Roll20)

Deployment Modes

Glyphoxa runs in four modes via --mode:

Mode	Description
`full` (default)	Single-process, self-hosted. No admin API, config from YAML.
`gateway`	Multi-tenant orchestrator. Admin API, bot management, session routing via gRPC.
`worker`	Voice pipeline executor. Receives sessions from gateway, connects to Discord voice.
`mcp-gateway`	Shared MCP tool server for workers. Stateless tools over HTTP.

🏗️ Architecture

┌──────────────────────────────────────────────────────────┐
│                    Audio Transport                       │
│              (Discord / WebRTC / Custom)                 │
├────────────────────┬─────────────────────────────────────┤
│   Audio In (VAD)   │          Audio Out (Mixer)          │
├────────────────────┴─────────────────────────────────────┤
│              Agent Orchestrator + Router                  │
│    ┌─────────┐  ┌─────────┐  ┌─────────┐                │
│    │ NPC #1  │  │ NPC #2  │  │ NPC #3  │  ...           │
│    └────┬────┘  └────┬────┘  └────┬────┘                │
├─────────┴────────────┴────────────┴──────────────────────┤
│                  Voice Engines                            │
│    Cascaded (STT→LLM→TTS) │ S2S (Gemini/OpenAI) │ ⚠Cascade │
├──────────────────────────────────────────────────────────┤
│   Memory Subsystem          │    MCP Tool Execution      │
│  ┌─────┐ ┌─────┐ ┌─────┐  │  ┌──────┐ ┌──────┐         │
│  │ Log │ │ Vec │ │Graph│  │  │ Dice │ │Rules │ ...      │
│  └─────┘ └─────┘ └─────┘  │  └──────┘ └──────┘         │
└──────────────────────────────────────────────────────────┘

🚀 Quick Start

Prerequisites

Go 1.26+ with CGo enabled
libopus — apt install libopus-dev · pacman -S opus · brew install opus
ONNX Runtime — from onnxruntime releases (for Silero VAD)
libdave — make dave-libs (for Discord DAVE E2EE voice encryption)

Build & Run

git clone https://github.com/MrWong99/glyphoxa.git
cd glyphoxa

# Build
make build

# Run
./bin/glyphoxa --config config.yaml

Development

# Run tests with race detector (459 tests)
make test

# Full pre-commit check (fmt + vet + test)
make check

🔌 Provider Support

Component	Providers
STT	Deepgram Nova-3, whisper.cpp (local)
LLM	OpenAI, Anthropic, Google Gemini, Ollama (local) — via any-llm-go
TTS	ElevenLabs, Coqui XTTS (local)
S2S	Gemini Live, OpenAI Realtime
Embeddings	OpenAI, Ollama (local)
Audio	Discord, WebRTC
Memory	PostgreSQL + pgvector

⚡ Performance Targets

Metric	Target	Hard Limit
Mouth-to-ear latency	< 1.2s	2.0s
STT time-to-first-token	< 300ms	500ms
LLM time-to-first-token	< 400ms	800ms
TTS time-to-first-byte	< 200ms	500ms
Concurrent NPC voices	≥ 3	≥ 1
Hot memory assembly	< 50ms	< 150ms

📦 Project Structure

glyphoxa/
├── cmd/glyphoxa/          # Entry point (--mode=full|gateway|worker|mcp-gateway)
├── internal/
│   ├── agent/             # NPC agents, orchestrator, router, address detection
│   ├── config/            # Configuration, tenant context, hot-reload
│   ├── engine/            # Voice engines (S2S wrapper, sentence cascade)
│   ├── entity/            # Entity management (CRUD, YAML, VTT import)
│   ├── gateway/           # Admin API, bot manager, session orchestrator, gRPC
│   ├── health/            # Health probes (/healthz, /readyz)
│   ├── hotctx/            # Hot context assembly and formatting
│   ├── mcp/               # MCP host, bridge, budget tiers, built-in tools
│   ├── observe/           # Metrics, traces, structured logging (OTel)
│   ├── resilience/        # Circuit breaker, retry patterns
│   ├── session/           # Voice pipeline lifecycle (Runtime, WorkerHandler)
│   └── transcript/        # Transcript correction pipeline
├── pkg/
│   ├── audio/             # Platform + Connection interfaces, mixer, WebRTC
│   ├── memory/            # Store interface, PostgreSQL + pgvector, knowledge graph
│   └── provider/          # LLM, STT, TTS, S2S, VAD, Embeddings interfaces + impls
├── deploy/helm/           # Kubernetes Helm chart (gateway, worker, MCP gateway)
├── docs/design/           # Architecture and design documents
├── proto/                 # gRPC service definitions
├── research/              # Research notes
└── configs/               # Example configuration files

📖 Documentation

Comprehensive guides for developers and contributors — see the full documentation index.

Guide	Description
Getting Started	Prerequisites, build, first run
Architecture	System layers, data flow, key packages
Configuration	Complete config field reference
Providers	Provider system, adding new providers
NPC Agents	NPC definition, entities, campaigns
Memory	3-layer memory system
MCP Tools	Tool system, building custom tools
Audio Pipeline	Audio flow, VAD, engine types
Commands	Discord slash and voice commands
Deployment	Docker Compose, Kubernetes / Helm, production setup
Multi-Tenant	Gateway, admin API, tenant model, usage tracking
Observability	Metrics, Grafana, health endpoints
Testing	Test conventions and patterns
Troubleshooting	Common issues and debugging

📚 Design Documents

Document	Description
Overview	Vision, goals, product principles
Architecture	System layers and data flow
Providers	LLM, STT, TTS, Audio platform interfaces
Memory	Hybrid memory system and knowledge graph
MCP Tools	Tool integration and performance budgets
Sentence Cascade	⚠️ Dual-model cascade (experimental)
NPC Agents	Agent design and multi-NPC orchestration
Technology	Technology decisions and latency budget
Roadmap	Development phases
Knowledge Graph	L3 graph schema and query patterns

🤝 Contributing

See CONTRIBUTING.md for development setup, code style, and workflow guidelines.

Bugs → Bug Report
Features → Feature Request
Security → SECURITY.md

📄 License

Directories ¶

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL

Path	Synopsis
cmd
glyphoxa command Command glyphoxa is the main entry point for the Glyphoxa voice AI server.	Command glyphoxa is the main entry point for the Glyphoxa voice AI server.
gen
glyphoxa/v1
internal
agent Package agent defines the NPCAgent and Router interfaces, along with the identity and scene-context types used by the orchestrator to drive NPC behaviour.	Package agent defines the NPCAgent and Router interfaces, along with the identity and scene-context types used by the orchestrator to drive NPC behaviour.
agent/mock Package mock provides in-memory mock implementations of agent.NPCAgent and agent.Router for use in unit tests.	Package mock provides in-memory mock implementations of agent.NPCAgent and agent.Router for use in unit tests.
agent/npcstore Package npcstore provides persistent storage and management for NPC definitions.	Package npcstore provides persistent storage and management for NPC definitions.
agent/orchestrator Package orchestrator provides an agent.Router implementation that manages NPC agents within a Glyphoxa session.	Package orchestrator provides an agent.Router implementation that manages NPC agents within a Glyphoxa session.
app Package app wires all Glyphoxa subsystems into a running application.	Package app wires all Glyphoxa subsystems into a running application.
config Package config provides the configuration schema, loader, and provider registry for the Glyphoxa voice AI system.	Package config provides the configuration schema, loader, and provider registry for the Glyphoxa voice AI system.
discord Package discord provides the Discord bot layer for Glyphoxa.	Package discord provides the Discord bot layer for Glyphoxa.
discord/commands Package commands implements Discord slash command handlers for the Glyphoxa DM experience.	Package commands implements Discord slash command handlers for the Glyphoxa DM experience.
discord/voicecmd Package voicecmd implements keyword detection on STT finals for DM-only voice shortcuts.	Package voicecmd implements keyword detection on STT finals for DM-only voice shortcuts.
engine Package engine defines the VoiceEngine interface and its supporting types.	Package engine defines the VoiceEngine interface and its supporting types.
engine/cascade Package cascade implements an experimental dual-model sentence cascade engine.	Package cascade implements an experimental dual-model sentence cascade engine.
engine/mock Package mock provides an in-memory mock implementation of engine.VoiceEngine for use in unit tests.	Package mock provides an in-memory mock implementation of engine.VoiceEngine for use in unit tests.
engine/s2s Package s2s provides a engine.VoiceEngine implementation that wraps an s2s.Provider, bridging the turn-based VoiceEngine.Process API with the streaming S2S session interface.	Package s2s provides a engine.VoiceEngine implementation that wraps an s2s.Provider, bridging the turn-based VoiceEngine.Process API with the streaming S2S session interface.
entity Package entity provides pre-session entity management for Glyphoxa.	Package entity provides pre-session entity management for Glyphoxa.
feedback Package feedback provides a simple feedback storage layer for the Glyphoxa closed alpha.	Package feedback provides a simple feedback storage layer for the Glyphoxa closed alpha.
gateway Package gateway provides the gateway-mode components for multi-tenant Glyphoxa deployments: the internal admin API, bot management, and session orchestration.	Package gateway provides the gateway-mode components for multi-tenant Glyphoxa deployments: the internal admin API, bot management, and session orchestration.
gateway/grpctransport Package grpctransport provides gRPC-backed implementations of the gateway contracts for distributed mode (--mode=gateway and --mode=worker).	Package grpctransport provides gRPC-backed implementations of the gateway contracts for distributed mode (--mode=gateway and --mode=worker).
gateway/local Package local provides in-process implementations of the gateway contracts for --mode=full.	Package local provides in-process implementations of the gateway contracts for --mode=full.
gateway/sessionorch Package sessionorch provides session lifecycle orchestration for multi-tenant Glyphoxa deployments.	Package sessionorch provides session lifecycle orchestration for multi-tenant Glyphoxa deployments.
gateway/usage Package usage provides per-tenant usage tracking and quota enforcement.	Package usage provides per-tenant usage tracking and quota enforcement.
health Package health provides HTTP health and readiness check handlers.	Package health provides HTTP health and readiness check handlers.
hotctx Package hotctx assembles the always-injected "hot" context for every NPC LLM call in the Glyphoxa voice AI pipeline.	Package hotctx assembles the always-injected "hot" context for every NPC LLM call in the Glyphoxa voice AI pipeline.
mcp Package mcp defines the interface for a Model Context Protocol (MCP) host.	Package mcp defines the interface for a Model Context Protocol (MCP) host.
mcp/bridge Package bridge wires MCP tools into an S2S voice session.	Package bridge wires MCP tools into an S2S voice session.
mcp/mcphost Package mcphost provides a concrete implementation of the mcp.Host interface.	Package mcphost provides a concrete implementation of the mcp.Host interface.
mcp/mock Package mock provides an in-memory test double for the MCP mcp.Host interface.	Package mock provides an in-memory test double for the MCP mcp.Host interface.
mcp/tier Package tier provides a lightweight heuristic-based budget tier selector for MCP tool budgets in Glyphoxa voice sessions.	Package tier provides a lightweight heuristic-based budget tier selector for MCP tool budgets in Glyphoxa voice sessions.
mcp/tools Package tools defines the shared Tool type used by all built-in MCP tool packages in Glyphoxa.	Package tools defines the shared Tool type used by all built-in MCP tool packages in Glyphoxa.
mcp/tools/diceroller Package diceroller provides built-in MCP tools for resolving dice rolls and random table lookups in a TTRPG session.	Package diceroller provides built-in MCP tools for resolving dice rolls and random table lookups in a TTRPG session.
mcp/tools/fileio Package fileio provides built-in MCP tools for sandboxed file reading and writing.	Package fileio provides built-in MCP tools for sandboxed file reading and writing.
mcp/tools/memorytool Package memorytool provides built-in MCP tools that expose Glyphoxa's three-layer memory architecture to NPC agents.	Package memorytool provides built-in MCP tools that expose Glyphoxa's three-layer memory architecture to NPC agents.
mcp/tools/ruleslookup Package ruleslookup provides built-in MCP tools for searching and retrieving game rules from an embedded D&D 5e SRD dataset.	Package ruleslookup provides built-in MCP tools for searching and retrieving game rules from an embedded D&D 5e SRD dataset.
observe Package observe provides application-wide observability primitives for Glyphoxa: OpenTelemetry metrics, distributed tracing, structured logging, and HTTP middleware that ties them together.	Package observe provides application-wide observability primitives for Glyphoxa: OpenTelemetry metrics, distributed tracing, structured logging, and HTTP middleware that ties them together.
resilience Package resilience provides circuit breaker and provider failover primitives.	Package resilience provides circuit breaker and provider failover primitives.
session Package session provides session lifecycle management for Glyphoxa NPC agents.	Package session provides session lifecycle management for Glyphoxa NPC agents.
transcript Package transcript defines the transcript correction pipeline used by Glyphoxa to fix STT errors in domain-specific vocabulary.	Package transcript defines the transcript correction pipeline used by Glyphoxa to fix STT errors in domain-specific vocabulary.
transcript/llmcorrect Package llmcorrect implements a language-model-based transcript correction stage that resolves entity misspellings not caught by the phonetic matcher.	Package llmcorrect implements a language-model-based transcript correction stage that resolves entity misspellings not caught by the phonetic matcher.
transcript/phonetic Package phonetic implements the [transcript.PhoneticMatcher] interface using Double Metaphone phonetic encoding combined with Jaro-Winkler string similarity for ranked candidate selection.	Package phonetic implements the [transcript.PhoneticMatcher] interface using Double Metaphone phonetic encoding combined with Jaro-Winkler string similarity for ranked candidate selection.
pkg
audio Package audio defines the interfaces and types for audio platform connectivity and stream management within Glyphoxa.	Package audio defines the interfaces and types for audio platform connectivity and stream management within Glyphoxa.
audio/discord Package discord provides an audio.Platform implementation backed by Discord voice channels via the disgoorg/disgo library.	Package discord provides an audio.Platform implementation backed by Discord voice channels via the disgoorg/disgo library.
audio/mixer Package mixer provides a concrete audio.Mixer implementation backed by a priority queue.	Package mixer provides a concrete audio.Mixer implementation backed by a priority queue.
audio/mock Package mock provides in-memory mock implementations of the audio.Platform, audio.Connection, and audio.Mixer interfaces for use in unit tests.	Package mock provides in-memory mock implementations of the audio.Platform, audio.Connection, and audio.Mixer interfaces for use in unit tests.
audio/webrtc Package webrtc provides an audio.Platform implementation backed by WebRTC via pion/webrtc.	Package webrtc provides an audio.Platform implementation backed by WebRTC via pion/webrtc.
memory Package memory defines the three-layer memory architecture used by Glyphoxa NPC agents.	Package memory defines the three-layer memory architecture used by Glyphoxa NPC agents.
memory/export Package export provides campaign export and import as .tar.gz archives.	Package export provides campaign export and import as .tar.gz archives.
memory/mock Package mock provides in-memory test doubles for the memory layer interfaces.	Package mock provides in-memory test doubles for the memory layer interfaces.
memory/postgres Package postgres provides a PostgreSQL-backed implementation of the three-layer Glyphoxa memory architecture (L1 session log, L2 semantic index, L3 knowledge graph).	Package postgres provides a PostgreSQL-backed implementation of the three-layer Glyphoxa memory architecture (L1 session log, L2 semantic index, L3 knowledge graph).
provider/embeddings Package embeddings defines the Provider interface for vector embedding backends.	Package embeddings defines the Provider interface for vector embedding backends.
provider/embeddings/mock Package mock provides a test double for the embeddings.Provider interface.	Package mock provides a test double for the embeddings.Provider interface.
provider/embeddings/ollama Package ollama provides an embeddings provider backed by a local Ollama server.	Package ollama provides an embeddings provider backed by a local Ollama server.
provider/embeddings/openai Package openai provides an embeddings provider backed by the OpenAI API.	Package openai provides an embeddings provider backed by the OpenAI API.
provider/llm Package llm defines the Provider interface for Large Language Model backends.	Package llm defines the Provider interface for Large Language Model backends.
provider/llm/anyllm Package anyllm provides a universal LLM provider backed by github.com/mozilla-ai/any-llm-go, a unified multi-provider interface that supports OpenAI, Anthropic, Gemini, Ollama, DeepSeek, Mistral, Groq, and more.	Package anyllm provides a universal LLM provider backed by github.com/mozilla-ai/any-llm-go, a unified multi-provider interface that supports OpenAI, Anthropic, Gemini, Ollama, DeepSeek, Mistral, Groq, and more.
provider/llm/mock Package mock provides a test double for the llm.Provider interface.	Package mock provides a test double for the llm.Provider interface.
provider/s2s Package s2s defines the Provider interface for Speech-to-Speech (S2S) backends.	Package s2s defines the Provider interface for Speech-to-Speech (S2S) backends.
provider/s2s/gemini Package gemini implements the s2s.Provider interface for Google's Gemini Live API.	Package gemini implements the s2s.Provider interface for Google's Gemini Live API.
provider/s2s/mock Package mock provides test doubles for the s2s package interfaces.	Package mock provides test doubles for the s2s package interfaces.
provider/s2s/openai Package openai implements the s2s.Provider interface for OpenAI's Realtime API.	Package openai implements the s2s.Provider interface for OpenAI's Realtime API.
provider/stt Package stt defines the Provider interface for Speech-to-Text backends.	Package stt defines the Provider interface for Speech-to-Text backends.
provider/stt/deepgram Package deepgram provides a Deepgram-backed STT provider using the Deepgram streaming WebSocket API.	Package deepgram provides a Deepgram-backed STT provider using the Deepgram streaming WebSocket API.
provider/stt/elevenlabs Package elevenlabs provides an ElevenLabs Scribe v2 Realtime STT provider using the ElevenLabs streaming WebSocket API.	Package elevenlabs provides an ElevenLabs Scribe v2 Realtime STT provider using the ElevenLabs streaming WebSocket API.
provider/stt/mock Package mock provides test doubles for the stt package interfaces.	Package mock provides test doubles for the stt package interfaces.
provider/stt/whisper Package whisper provides a local whisper.cpp-backed STT provider.	Package whisper provides a local whisper.cpp-backed STT provider.
provider/tts Package tts defines the Provider interface for Text-to-Speech backends.	Package tts defines the Provider interface for Text-to-Speech backends.
provider/tts/coqui Package coqui provides a local Coqui TTS-backed TTS provider that connects to either a Coqui XTTS v2 server or a standard Coqui TTS server via its REST API.	Package coqui provides a local Coqui TTS-backed TTS provider that connects to either a Coqui XTTS v2 server or a standard Coqui TTS server via its REST API.
provider/tts/elevenlabs Package elevenlabs provides an ElevenLabs-backed TTS provider using the ElevenLabs streaming WebSocket API.	Package elevenlabs provides an ElevenLabs-backed TTS provider using the ElevenLabs streaming WebSocket API.
provider/tts/mock Package mock provides a test double for the tts.Provider interface.	Package mock provides a test double for the tts.Provider interface.
provider/vad Package vad defines the Engine interface for Voice Activity Detection backends.	Package vad defines the Engine interface for Voice Activity Detection backends.
provider/vad/energy Package energy implements a pure-Go, energy-based Voice Activity Detection engine with no external dependencies.	Package energy implements a pure-Go, energy-based Voice Activity Detection engine with no external dependencies.
provider/vad/mock Package mock provides test doubles for the vad package interfaces.	Package mock provides test doubles for the vad package interfaces.
provider/vad/silero Package silero implements the vad.Engine interface using the Silero VAD v5 ONNX model via the yalue/onnxruntime_go binding.	Package silero implements the vad.Engine interface using the Silero VAD v5 ONNX model via the yalue/onnxruntime_go binding.