llm

go-llm

A multi-provider LLM tool and SDK for Go.

Interact with large language models from Google Gemini, Anthropic Claude, and Mistral through a unified CLI, HTTP API server, or Go SDK. Supports stateless completion (ask), stateful multi-turn conversations (chat), embeddings, tool use, MCP integration, and a Telegram bot.

Features

• Multi-Provider: Unified interface across Google Gemini, Anthropic Claude, and Mistral
• CLI: Interactive terminal UI chat with markdown rendering, single-shot ask, and embedding generation
• HTTP API Server: REST endpoints for models, sessions, tools, chat, ask, and embeddings with SSE streaming
• Session Management: Stateful multi-turn conversations with persistent storage (file-backed JSON)
• Tool Use: Extensible tool framework with built-in integrations for Home Assistant, NewsAPI, and WeatherAPI
• MCP Server: Model Context Protocol (stdio JSON-RPC 2.0) server exposing registered tools and prompts
• Telegram Bot: Full-featured bot with per-conversation sessions, file/image attachments, and slash commands
• Attachments: Send files, images, and URLs as context in chat and ask
• Agents: Reusable, versioned task definitions with input/output schemas, Go templates, and tool selection
• Structured Output: JSON schema-based structured output for ask and agents
• Thinking/Reasoning: Extended thinking support with configurable budgets (Anthropic, Gemini)
• OpenTelemetry: Distributed tracing support for the HTTP API

Quick Start

Server

The server provides the REST API for all providers and tools. Run it with Docker:

docker volume create go-llm
docker run -d --name go-llm \
  -v go-llm:/data -p 8085:8085 \
  -e GEMINI_API_KEY="your-key" \
  ghcr.io/mutablelogic/llm run

Client

The client-only CLI can be downloaded from the releases page, or use go install -tags client github.com/mutablelogic/go-llm/cmd/llm@latest. It does not include the server or the Telegram bot. Point it at a running server:

export LLM_ADDR="localhost:8085"

# List providers and models
llm providers
llm models

# Single-shot completion
llm ask --model gemini-2.0-flash "Explain KV cache in two sentences"

# Interactive chat (launches terminal UI)
llm chat --model gemini-2.0-flash

# Single-shot chat with session persistence
llm chat --model gemini-2.0-flash "What is the capital of France?"

# Generate embeddings
llm embedding --model text-embedding-004 "Hello, world"

Telegram Bot

The Telegram bot runs as a sidecar to the server, providing chat access via Telegram. It requires a bot token from @BotFather:

docker run -d --name go-llm-telegram \
  -e LLM_ADDR="your-server:8085" \
  -e TELEGRAM_TOKEN="your-bot-token" \
  ghcr.io/mutablelogic/llm telegram --model gemini-2.0-flash

Server Configuration

Providers

Providers are sources of models and generation capabilities. Some providers may support additional features like thinking/reasoning, tool use and embeddings. API keys for access to provider models are configured on the server via flags or environment variables:

Tools

Tools are external functions that can be called by the agent during generation. They are registered with the agent and exposed via the API and MCP server. Tools can be configured with flags or environment variables as needed for authentication or connection details. The following tools are included as examples of how to build tool integrations with the SDK:

HTTP & TLS

Observability

Sessions and defaults are stored in $XDG_CACHE_HOME (or system temp if unset).

CLI Commands

The llm command-line tool can connect to any running go-llm server (set LLM_ADDR or --http.addr to point at it) and provides commands for generating text, managing models and sessions, inspecting tools, and running the server itself.

Generate

Model

Session

Sessions represent stateful conversations with context and memory. They can be managed via the CLI or API, and are persisted on the server for later retrieval.

Tool

Agent

Agents are reusable, versioned task definitions that bundle a model, system prompt, optional input/output JSON schemas, Go template, and tool list into a single unit. They are defined as markdown files with YAML front matter and managed via the CLI or API. See the Agent Definition Files documentation for the full file format, field reference, template syntax, and examples.

Running an Agent

# Create the agent from a file
llm create-agent agents/summarizer.md

# Run with JSON input
llm run-agent summarizer --input '{"text": "The quick brown fox..."}'

# Run with file attachments
llm run-agent caption --file photo.jpg

# Run with URL attachment
llm run-agent summarizer --url https://example.com/article

# Keep the session after completion (for inspection or follow-up)
llm run-agent summarizer --no-delete --input '{"text": "..."}'

The run-agent command creates a temporary session, sends the templated message, and returns the structured JSON result. By default the session is deleted after completion; use --no-delete to persist it.

Server

Use llm --help or llm <command> --help for full options.

Interactive Chat

The chat command launches a terminal UI with markdown rendering, streaming responses, and slash commands:

Slash Commands

Development

Architecture

%%{init: {"flowchart": {"htmlLabels": false}} }%%
flowchart LR
    subgraph Clients["Clients"]
        direction TB
        CLI["`**cmd/llm**
        CLI Tool`"]
        TUI["`**pkg/ui/bubbletea**
        Terminal UI`"]
        SDK["`**pkg/httpclient**
        SDK Client`"]
        Telegram["`**pkg/ui/telegram**
        Telegram Bot`"]
    end

    subgraph Server["go-llm Server"]
        direction TB
        API["`**pkg/httphandler**
        REST API`"]
        Agent["`**pkg/manager**
        Manager`"]
        AgentDef["`**pkg/agent**
        Agent Parsing`"]
        Sessions["`**pkg/store**
        Memory, File`"]
        Tools["`**pkg/tool**
        Tool Registry`"]
        MCP["`**pkg/mcp**
        MCP Server`"]
    end

    subgraph Providers["LLM Providers"]
        direction TB
        Gemini["`**pkg/provider/google**
        Google Gemini`"]
        Claude["`**pkg/provider/anthropic**
        Anthropic Claude`"]
        Mistral["`**pkg/provider/mistral**
        Mistral`"]
        Eliza["`**pkg/provider/eliza**
        ELIZA (mock)`"]
    end

    subgraph ToolImpls["Example Tools"]
        direction TB
        HA["`**pkg/homeassistant**
        Home Assistant`"]
        News["`**pkg/newsapi**
        NewsAPI`"]
        Weather["`**pkg/weatherapi**
        WeatherAPI`"]
    end

    CLI --> API
    TUI --> API
    SDK --> API
    Telegram --> API
    API --> Agent
    Agent --> AgentDef
    Agent --> Sessions
    Agent --> Tools
    Agent --> Gemini
    Agent --> Claude
    Agent --> Mistral
    Agent --> Eliza
    Tools --> HA
    Tools --> News
    Tools --> Weather
    MCP --> Tools

SDK Example

The Go SDK provides interfaces for building LLM-powered applications:

package main

import (
    "context"
    "fmt"
    "log"

    llm "github.com/mutablelogic/go-llm"
    "github.com/mutablelogic/go-llm/pkg/provider/google"
)

func main() {
    ctx := context.Background()

    // Create a provider
    client, err := google.New(ctx, "your-api-key")
    if err != nil {
        log.Fatal(err)
    }

    // Get a model
    model, err := client.GetModel(ctx, "gemini-2.0-flash")
    if err != nil {
        log.Fatal(err)
    }

    // Stateless completion
    gen := model.(llm.Generator)
    response, err := gen.WithoutSession(ctx, "Explain KV cache in two sentences")
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(response)
}

Key Packages

Core Interfaces

Building

git clone https://github.com/mutablelogic/go-llm.git
cd go-llm
make

Build the client-only CLI (no server or Telegram support):

make llm-client

Makefile Targets

Cross-compilation is supported via OS and ARCH variables:

OS=linux ARCH=arm64 make

Contributing & License

Please file issues and feature requests in GitHub Issues. Licensed under Apache 2.0.