talkxtyper

TalkXTyper

TalkXTyper is a desktop application that will, on command, record your voice, transcribe it using the OpenAI transcription API, and "type" it to your computer. It is activated with a global hotkey so that you do not lose focus on the area you're typing into.

Rationale

There are a few transcription tools out there, but I wanted to create my own so I could explore different ideas based around my own workflow.

Although modern transcription models are very good, they do not automatically know what is on the screen. For example, if you are coding and want to reference a variable on the screen named my_variable, saying "my variable" may produce "My variable" instead of the symbol on the screen.

Attempts

1. Send a desktop screenshot to the context model

   • Idea: take and send screenshot of the desktop while audio is being recorded, send image to gpt-4o to ask it to extract relevant textual features from the image. Combine the extracted information with the transcription output to attempt to fix the transcription to match text on the screen.
     • Result: vision processing can make the typing experience too slow
   • Use Claude Sonnet 3.5, it appears to be much faster with image processing

2. Provide context to the transcription model The gpt-transcribe API accepts a free-form prompt plus keyword and language hints. TalkXTyper passes available screen, Neovim, or tmux context into the transcription request, derives literal technical keyword hints, and retains a second repair pass for context-specific corrections. Submitted keywords are visible in the transcription history page.

3. Extract text from running app Idea: Query what the currently focused app is, then have custom code to extract the text from the screen.

   • Implement text extraction from nvim using the nvim remote API
   • Explore extracting text from browser. (Consider a browser extension)

Hotkeys

TalkXTyper registers the following global hotkeys:

• Alt+B: Toggle recording. Press once to start recording, press again to stop recording and begin transcription.
• Alt+C: Abort the current task. This cancels the operation regardless of state, so it works both while recording and while a transcription is already in flight. Aborting discards the result and types nothing.

These actions are also available from the systray menu ("Record and Transcribe" and "Abort Recording").

The systray's Transcription mode submenu switches between:

• Buffered — gpt-transcribe: records the complete utterance and uploads it after you stop.
• Live — gpt-live-transcribe: starts recording immediately while a Realtime transcription session is opened in the background. Audio buffers locally until the session is ready, then streams as 24 kHz PCM, and the transcript is typed incrementally as it arrives. Stopping commits the audio turn and types any remaining text from the final transcript.

The selected mode is saved in the configuration file. Context prompts, keyword hints, language hints, and MP3 history are used in both modes. The optional repair pass only runs in buffered mode: live mode types text as you speak, so it cannot be revised afterwards.

Configuration

The configuration for TalkXTyper is stored in a JSON file located in your user configuration directory. The file is named talkxtyper-config.json.

Configuration Options

• OpenAIKey: Your API key for the OpenAI API.
• TranscriptionMode: "buffered" (the default) or "live".
• IncludeScreen: A boolean value indicating whether to analyze the screen to augment the transcription. The config file will be updated automatically if you change this value in the program.
• IncludeNvim: A boolean value indicating whether to analyze the screen to augment the transcription.

Web interface

ListenAddress can be specified in the config file to enable the web interface. The web interface includes some experimental functionality. The web interface is not enabled by default.

Eg. Setting ListenAddress to "localhost:9898" will make the web interface accessible at http://localhost:9898.

SECURITY NOTE: The web interface adds a HTTP API for controlling recording and transcribing, in addition to taking screenshots of the desktop. Don't leave it running if you don't need it.

The web interface exposes a way to review transcription history via /history and listen to the audio files that were recorded. You can use this to debug if recording is working as expected.

Installation

To install TalkXTyper, you will need to have Go installed. Run the following command:

go install github.com/leafo/talkxtyper@latest

This project has only been tested on Linux, but it uses cross-platform libraries, so it should work on other platforms.

License

This project is licensed under the MIT License. See the LICENSE file for details.