llm

package module
v0.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 10, 2026 License: MIT Imports: 3 Imported by: 0

README

llm

Small, native, per-provider model adapters for agentcore / jess agents, behind one domain-named port: llm.LLM.

Why

agentcore.ChatModel is the interface an agent talks to. llm.LLM is that same contract under a name that belongs to your domain, so the rest of your code never imports a vendor SDK. Each provider adapter is:

  • Native — it speaks the provider's own API, not an OpenAI-compatibility shim layered over a different provider.
  • An anti-corruption layer — the only package allowed to import that provider's SDK, translating it to and from agentcore message / tool / stream-event types.

Add a provider by adding a subpackage; the rest of your agent doesn't change.

What it's not

Not a gateway. If you want one process that fronts every provider behind an OpenAI-compatible endpoint, with routing, fallbacks, load balancing, budgets, and a proxy server, that's LiteLLM or OpenRouter. llm is the opposite shape:

  • A library, not a server. No proxy, no gateway, no daemon. You import it; calls go straight from your process to the provider.
  • Native, not OpenAI-flattened. Each adapter speaks its provider's real API and can surface provider-specific behavior (Anthropic's thinking/effort), instead of collapsing everything to a lowest-common-denominator shape.
  • No orchestration baked in. Routing, fallback, retries, load balancing, response caching, cost and budget tracking are the agent's or caller's job, not this layer's.
  • No giant model registry. You add the one or two adapters you actually use; nothing else ships.

One port (llm.LLM), native adapters behind it. That's the whole scope.

Install

go get github.com/guygrigsby/llm

Requires Go 1.26+.

The port

type LLM interface {
	Generate(ctx context.Context, messages []agentcore.Message, tools []agentcore.ToolSpec, opts ...agentcore.CallOption) (*agentcore.LLMResponse, error)
	GenerateStream(ctx context.Context, messages []agentcore.Message, tools []agentcore.ToolSpec, opts ...agentcore.CallOption) (<-chan agentcore.StreamEvent, error)
	SupportsTools() bool
}

Anything satisfying llm.LLM is an agentcore.ChatModel, so it drops straight into jess.WithModel. agentcore types cross the boundary freely — they are the ubiquitous language jess is built on, not an isolated vendor. The isolated vendor is each provider's SDK, which only that provider's adapter imports.

Adapters

anthropic

github.com/guygrigsby/llm/anthropic — native Anthropic adapter on the official anthropic-sdk-go. Adaptive thinking + effort; never sends temperature/top_p/top_k. Full streaming and tool support.

m, err := anthropic.New(anthropic.Config{APIKey: key, Model: "claude-sonnet-5"})
if err != nil {
	return err
}
agent := jess.New(jess.WithModel(m) /* , ... */)
deepseek

github.com/guygrigsby/llm/deepseek — native DeepSeek adapter over its OpenAI-format chat-completions API, using only net/http (no SDK dependency). Built for cheap one-shot work such as summaries and extraction; tool-calling and true token streaming are not wired yet (GenerateStream returns the whole result as one terminal event). Wire those before using it as a primary conversational model.

m, err := deepseek.New(deepseek.Config{APIKey: key, Model: "deepseek-chat"})
kimi

github.com/guygrigsby/llm/kimi — native Kimi (Moonshot) adapter over its OpenAI-format chat-completions API, using only net/http (no SDK dependency). Full scope: real SSE token streaming and tool calling, so kimi-k3 works as a primary conversational model. Kimi-specific extensions beyond standard OpenAI are wired too: reasoning_effort (mapped from the call's thinking level, with the model's reasoning_content surfaced as a thinking block), partial mode (prefix continuation, triggered by "partial": true metadata on a trailing assistant message), and an EstimateTokens helper over the /tokenizers/estimate-token-count endpoint.

m, err := kimi.New(kimi.Config{APIKey: key, Model: "kimi-k3"})
if err != nil {
	return err
}
agent := jess.New(jess.WithModel(m) /* , ... */)
openrouter

github.com/guygrigsby/llm/openrouter, a native OpenRouter adapter over its OpenAI-format chat-completions API, using only net/http (no SDK dependency). One key, any OpenRouter model id per adapter (moonshotai/kimi-k3, anthropic/claude-opus-5, ...). Full scope: SSE token streaming and tool calling. OpenRouter extensions are wired: the reasoning request object with effort mapped from the call's thinking level; the model's reasoning surfaced as a thinking block; reasoning_details echoed back on assistant messages so the model keeps its chain of thought across tool calls. Usage accounting reaches the Meter with cost and cached prompt tokens.

m, err := openrouter.New(openrouter.Config{APIKey: key, Model: "moonshotai/kimi-k3", Title: "my-app"})
if err != nil {
	return err
}
agent := jess.New(jess.WithModel(m) /* , ... */)
aperture

github.com/guygrigsby/llm/aperture, a native adapter for Aperture, the Tailscale model gateway on the tailnet, over its OpenAI-format chat-completions API (the OpenRouter dialect, shared with openrouter through internal/chatwire). The tailnet authenticates the host rather than a bearer, so no API key is required (one can still be set for a route that asks); the endpoint defaults to https://ai.guy.ts.net/v1 and BaseURL overrides it. Model ids are Aperture's own, as its /v1/models lists them (anthropic/claude-sonnet-5.5, moonshotai/kimi-k3, qwen-27b, ...). Full scope matches openrouter: SSE token streaming, tool calling, the reasoning request object and reasoning_details echo, and usage accounting (cost included where the route prices) reaching the Meter labelled aperture.

m, err := aperture.New(aperture.Config{Model: "anthropic/claude-sonnet-5.5"})
if err != nil {
	return err
}
agent := jess.New(jess.WithModel(m) /* , ... */)
openaicompat

github.com/guygrigsby/llm/openaicompat, for any server whose native API is plain OpenAI chat completions: a user's own endpoint (vLLM, llama.cpp, Ollama, LM Studio, a hosted provider) that the caller can't know in advance. It sends only the shape every compatible server shares: max_tokens, reasoning_effort only when the call sets a thinking level, and no reasoning echo on assistant history. BaseURL and Model are required; APIKey is optional and agentcore.WithAPIKey overrides it per call, so one adapter can carry many users' keys. Transport carries every request, for a host that must restrict where user-supplied URLs can reach. SSE token streaming and tool calling; usage reaches the Meter labelled openaicompat when the server reports it.

m, err := openaicompat.New(openaicompat.Config{BaseURL: userURL, Model: userModel, Transport: restricted})
if err != nil {
	return err
}
resp, err := m.Generate(ctx, msgs, nil, agentcore.WithAPIKey(userKey))

License

MIT. See LICENSE.

Documentation

Overview

Package llm is the LLM port for jess/agentcore-based agents, plus a home for native, per-provider adapters that satisfy it.

LLM is the ubiquitous term for "the model the agent talks to." It mirrors the agentcore ChatModel contract, so anything satisfying LLM plugs directly into jess.WithModel — but naming it here keeps the vendor type out of consumers' domain language.

Each provider adapter lives in its own subpackage (llm/anthropic, and later llm/openai, ...) and is native: it speaks that provider's own API/SDK, not an OpenAI-compatible translation. An adapter is an anti-corruption layer — the only package allowed to import its provider's SDK — translating that SDK to and from agentcore message / tool / stream-event types.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type LLM

type LLM interface {
	Generate(ctx context.Context, messages []agentcore.Message, tools []agentcore.ToolSpec, opts ...agentcore.CallOption) (*agentcore.LLMResponse, error)
	GenerateStream(ctx context.Context, messages []agentcore.Message, tools []agentcore.ToolSpec, opts ...agentcore.CallOption) (<-chan agentcore.StreamEvent, error)
	SupportsTools() bool
}

LLM is the model port the agent consumes. It is the agentcore ChatModel contract under a domain name: any LLM is usable directly as an agent's model. agentcore types cross this boundary freely — they are the ubiquitous language jess is built on, not an isolated vendor. The isolated vendor is each provider's SDK, which only that provider's adapter imports.

type Meter added in v0.2.0

type Meter interface {
	Observe(Usage)
}

Meter observes per-call Usage. Set one on an adapter's Config to capture tokens and latency for every generation. Implementations must be safe for concurrent use — adapters may call Observe from multiple goroutines.

type MeterFunc added in v0.2.0

type MeterFunc func(Usage)

MeterFunc adapts a plain function to a Meter.

func (MeterFunc) Observe added in v0.2.0

func (f MeterFunc) Observe(u Usage)

Observe calls f.

type Usage added in v0.2.0

type Usage struct {
	Provider         string        `json:"provider"`
	Model            string        `json:"model"`
	PromptTokens     int           `json:"prompt_tokens"`
	CompletionTokens int           `json:"completion_tokens"`
	TotalTokens      int           `json:"total_tokens"`
	Latency          time.Duration `json:"latency"`
	// CacheReadTokens and CacheWriteTokens split out prompt-caching input tokens
	// so a Meter can price the cache tiers (read ~0.1x, write ~1.25x-2x base
	// input) and measure hit rate. Zero when the provider reports no caching or
	// none was requested. PromptTokens is the uncached-input remainder, so the
	// three are additive: total input = PromptTokens + CacheReadTokens + CacheWriteTokens.
	CacheReadTokens  int `json:"cache_read_tokens,omitempty"`
	CacheWriteTokens int `json:"cache_write_tokens,omitempty"`
	// ReasoningTokens is the part of CompletionTokens the model spent on
	// reasoning, for providers that break it out (OpenRouter's
	// completion_tokens_details.reasoning_tokens). It is a subset of
	// CompletionTokens, not an addition, so a Meter must not add it to the
	// total. Zero when the provider reports no split.
	ReasoningTokens int `json:"reasoning_tokens,omitempty"`
	// Cost is the provider-reported cost of the call in USD credits, for
	// providers that price per call (OpenRouter). Zero when not reported; a
	// Meter that prices from tokens ignores it.
	Cost float64 `json:"cost,omitempty"`
}

Usage reports the token counts and wall-clock latency of a single model call. Adapters populate it from the provider response and hand it to a Meter (when one is configured), so a consumer can record cost and latency without the adapter knowing anything about pricing.

Directories

Path Synopsis
Package anthropic is the provider-native Anthropic model adapter: the anti-corruption layer that is the ONLY package allowed to import github.com/anthropics/anthropic-sdk-go.
Package anthropic is the provider-native Anthropic model adapter: the anti-corruption layer that is the ONLY package allowed to import github.com/anthropics/anthropic-sdk-go.
Package aperture is the adapter for Aperture, the model gateway on the tailnet: an agentcore.ChatModel (llm.LLM) backed by its chat-completions API.
Package aperture is the adapter for Aperture, the model gateway on the tailnet: an agentcore.ChatModel (llm.LLM) backed by its chat-completions API.
Package deepseek is a native DeepSeek model adapter: an agentcore.ChatModel (llm.LLM) backed by DeepSeek's chat-completions API.
Package deepseek is a native DeepSeek model adapter: an agentcore.ChatModel (llm.LLM) backed by DeepSeek's chat-completions API.
internal
chatwire
Package chatwire is the OpenRouter-dialect chat-completions wire format: the standard OpenAI shape plus the routed-gateway extensions (the reasoning request object, the reasoning_details echo, cost-bearing usage) that OpenRouter's API defines and gateways that speak its dialect share.
Package chatwire is the OpenRouter-dialect chat-completions wire format: the standard OpenAI shape plus the routed-gateway extensions (the reasoning request object, the reasoning_details echo, cost-bearing usage) that OpenRouter's API defines and gateways that speak its dialect share.
openaiwire
Package openaiwire holds the OpenAI-shaped chat-completions wire format shared by every native adapter whose provider speaks that dialect (kimi, openrouter): the request and response structs, message and tool conversion, stop-reason mapping, the SSE emitter and the authenticated HTTP client.
Package openaiwire holds the OpenAI-shaped chat-completions wire format shared by every native adapter whose provider speaks that dialect (kimi, openrouter): the request and response structs, message and tool conversion, stop-reason mapping, the SSE emitter and the authenticated HTTP client.
Package kimi is the provider-native Kimi (Moonshot) model adapter: an agentcore.ChatModel (llm.LLM) backed by Kimi's chat-completions API.
Package kimi is the provider-native Kimi (Moonshot) model adapter: an agentcore.ChatModel (llm.LLM) backed by Kimi's chat-completions API.
Package openaicompat is the adapter for any server that speaks the plain OpenAI chat-completions API: a user's own endpoint (vLLM, llama.cpp, Ollama, LM Studio, a hosted provider) whose real identity this code never knows.
Package openaicompat is the adapter for any server that speaks the plain OpenAI chat-completions API: a user's own endpoint (vLLM, llama.cpp, Ollama, LM Studio, a hosted provider) whose real identity this code never knows.
Package openrouter is the provider-native OpenRouter model adapter: an agentcore.ChatModel (llm.LLM) backed by OpenRouter's chat-completions API.
Package openrouter is the provider-native OpenRouter model adapter: an agentcore.ChatModel (llm.LLM) backed by OpenRouter's chat-completions API.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL