llm

package
v0.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 7, 2026 License: MIT Imports: 24 Imported by: 0

Documentation

Overview

claude.go is the native Anthropic Messages provider. Auth is direct API-key only (ANTHROPIC_API_KEY via x-api-key header). bee identifies itself honestly as bee/0.1; impersonating first-party clients to access subscription pricing is out of scope.

Package llm defines the Provider interface and shared event types.

Concrete provider adapters (openai_compat.go, anthropic.go) live alongside and translate the internal types in github.com/elhenro/bee/internal/types to/from each provider's wire format. The rest of the codebase only depends on this interface.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ClearModelCache

func ClearModelCache()

ClearModelCache drops every cached entry. Tests call this between cases.

func ContextWindow

func ContextWindow(modelID string) int

ContextWindow returns a best-effort context length (in tokens) for the given model id. Live API-learned values win over the hardcoded table. Tries the exact id, then the trailing path segment so OpenRouter ids ("anthropic/claude-sonnet-4-6") resolve to the bare model name. Returns 0 when unknown — TUI treats that as "no fill indicator".

func IsQwen3HybridThinking

func IsQwen3HybridThinking(modelID string) bool

IsQwen3HybridThinking reports whether modelID names a Qwen3 model that supports the `/think` `/no_think` system-prompt toggle. Unlike the reasoning_effort wire path (covered by SupportsThinking), Qwen3 hybrid inference servers (omlx/lmstudio/ollama running qwen3-*-a3b, qwen3-235b, etc.) consume the toggle as a literal token in the prompt.

Excludes already-explicit thinking variants (qwq, qwen3-thinking, qwen3-reasoner): those models think unconditionally, no toggle needed.

Also excludes the coder family (qwen3-coder-*): those ship with reasoning disabled, emit no trace, and treat the toggle as a no-op. Injecting it just adds prompt noise and misreports the model as a thinker.

Heuristic: substring "qwen3" or "qwen-3" present AND no explicit thinking suffix AND not a coder variant. Matches sparse-MoE (a3b, a7b) and flagship (qwen3-235b) families.

func OllamaBaseURL

func OllamaBaseURL(baseURL string) string

OllamaBaseURL trims the trailing "/v1" segment from an openai-compat base URL so ollama's native paths (/api/show, /api/tags) resolve cleanly. Returns the input unchanged when no /v1 suffix is present.

func ProbeContextLength

func ProbeContextLength(ctx context.Context, name string, cfg config.ProviderConfig, modelID string) int

ProbeContextLength returns the live context window for modelID, learning it from the provider if not already in the cache. Order:

  1. ContextWindow cache (live + hardcoded) — short-circuits.
  2. Ollama native /api/show — POST {"name": modelID}, parse model_info.
  3. OpenAI-compat /v1/models — populates cache for any model the server advertises a context_length for (openrouter, some lm-studio versions).

Returns the learned value (also stored via RememberContextLength) or 0 when nothing could be determined. Safe to call concurrently and repeatedly — dedupes per (provider,model).

func ProbeOllamaContext

func ProbeOllamaContext(ctx context.Context, client *http.Client, baseURL, modelID string) (int, error)

ProbeOllamaContext POSTs to <baseURL>/api/show with {"name": modelID}, then extracts the model's real context length from either the `parameters` free-form string (`num_ctx <N>`) or the `model_info` map (any key ending in `.context_length`). Best-effort: returns (0, nil) when the endpoint exists but yields nothing usable, and (0, err) only on network/HTTP failures so callers can log without aborting.

func Qwen3ThinkingHint

func Qwen3ThinkingHint(t Thinking) string

Qwen3ThinkingHint maps a resolved Thinking level into the literal toggle token Qwen3 hybrid models consume. ThinkingOff / ThinkingLow → `/no_think` (skip reasoning trace entirely). ThinkingMedium / High / Max → `/think`. Empty string means "no toggle" — caller skips injection.

func RememberContextLength

func RememberContextLength(modelID string, n int)

RememberContextLength records a live-learned context window for modelID. Both the raw id and its trailing path segment are stored so future lookups resolve regardless of which form ContextWindow gets.

func ResetLiveContextLengths

func ResetLiveContextLengths()

ResetLiveContextLengths drops every learned entry. Tests call this.

func ResetProbed

func ResetProbed()

ResetProbed drops dedupe state. Tests call between cases.

func SetModelsHTTPClient

func SetModelsHTTPClient(c *http.Client)

SetModelsHTTPClient replaces the package HTTP client. Tests use this to route through an httptest server with a stricter timeout.

func SetProbeHTTPClient

func SetProbeHTTPClient(c *http.Client)

SetProbeHTTPClient swaps the package probe client. Tests route through httptest servers.

func SupportsThinking

func SupportsThinking(modelID string) bool

SupportsThinking reports whether modelID belongs to a family that honors a reasoning_effort / thinking-budget request. Best-effort: substring match against thinkingModelSubstrings on both the full id and trailing path segment. Unknown models return false → ThinkingAuto resolves to off so we don't break non-reasoning providers that choke on the field.

func SupportsVision

func SupportsVision(modelID string) bool

SupportsVision reports whether modelID belongs to a family that accepts image content blocks. Substring match against visionModelSubstrings on both the full id and trailing path segment. Unknown models return false → the loop's vision fallback kicks in (describe-then-inject) when one is configured.

func ThinkingApplies

func ThinkingApplies(modelID string) bool

ThinkingApplies reports whether modelID uses a reasoning/thinking mechanism in any form — the reasoning_effort wire field (SupportsThinking families) or the Qwen3 hybrid /think system-prompt token. False means an effort level is a pure no-op for the model (e.g. qwen3-coder, plain instruct models): callers use this to avoid showing or sending effort that the model can't act on.

func ThinkingBudget

func ThinkingBudget(t Thinking) int

ThinkingBudget maps a level to a token budget for thinking-enabled providers. Off returns 0 → caller should omit the thinking field.

Types

type AnthropicProvider

type AnthropicProvider struct {
	// contains filtered or unexported fields
}

AnthropicProvider is a stub. The full native implementation is in claude.go. Most users should route through OpenAI-compatible paths or use the anthropic-messages wire_api config option.

func NewAnthropic

func NewAnthropic(apiKeyEnv, baseURL string) *AnthropicProvider

NewAnthropic builds a stub provider. apiKeyEnv is the env var for the bearer token; baseURL defaults to https://api.anthropic.com/v1 when empty.

func (*AnthropicProvider) Name

func (p *AnthropicProvider) Name() string

Name returns the static provider identifier.

func (*AnthropicProvider) Stream

func (p *AnthropicProvider) Stream(ctx context.Context, req Request) (<-chan Event, error)

Stream is intentionally not implemented on this stub. Use the native path in claude.go or configure the provider with wire_api = "anthropic-messages" via openai_compat.go.

type ChatGPTConfig

type ChatGPTConfig struct {
	// Name shows up in logs and Provider.Name(). Defaults to "chatgpt".
	Name string
	// BaseURL e.g. "https://chatgpt.com/backend-api/codex". Path "/responses"
	// is appended automatically.
	BaseURL string
	// ClientID is the OAuth client id used to refresh tokens on 401.
	ClientID string
	// TokenEndpoint is the OAuth token URL for refresh.
	TokenEndpoint string
	// AccountIDHeader is the HTTP header that carries the per-account id
	// (e.g. "chatgpt-account-id"). Empty disables injection.
	AccountIDHeader string
	// HTTPClient is overridable for tests. Defaults to a long-timeout client.
	HTTPClient *http.Client
	// ExtraHeaders are merged into every request.
	ExtraHeaders map[string]string
}

ChatGPTConfig drives an OpenAI Responses-API provider with subscription auth. Use this for chatgpt.com/backend-api/codex (Plus/Pro/Team plans).

type ChatGPTProvider

type ChatGPTProvider struct {
	// contains filtered or unexported fields
}

ChatGPTProvider implements Provider against OpenAI's Responses API on the ChatGPT-subscription backend.

func NewChatGPT

func NewChatGPT(cfg ChatGPTConfig) *ChatGPTProvider

NewChatGPT builds a provider.

func (*ChatGPTProvider) Name

func (p *ChatGPTProvider) Name() string

Name returns the configured display name.

func (*ChatGPTProvider) Stream

func (p *ChatGPTProvider) Stream(ctx context.Context, req Request) (<-chan Event, error)

Stream issues a /responses call and emits Events on the returned channel.

type ClaudeConfig

type ClaudeConfig struct {
	// Name shows up in logs and Provider.Name(). Defaults to "claude".
	Name string
	// BaseURL e.g. "https://api.anthropic.com/v1". The "/messages" path is
	// appended automatically.
	BaseURL string
	// EnvKey is the API-key env var (e.g. ANTHROPIC_API_KEY).
	EnvKey string
	// HTTPClient is overridable for tests. Defaults to a long-timeout client.
	HTTPClient *http.Client
	// ExtraHeaders are merged into every request.
	ExtraHeaders map[string]string
}

ClaudeConfig configures the provider. API-key only — the Bearer/OAuth subscription path was removed.

type ClaudeProvider

type ClaudeProvider struct {
	// contains filtered or unexported fields
}

ClaudeProvider implements Provider against Anthropic's native Messages API.

func NewClaude

func NewClaude(cfg ClaudeConfig) *ClaudeProvider

NewClaude builds a provider.

func (*ClaudeProvider) Name

func (p *ClaudeProvider) Name() string

Name returns the configured display name.

func (*ClaudeProvider) Stream

func (p *ClaudeProvider) Stream(ctx context.Context, req Request) (<-chan Event, error)

Stream issues a /messages call and emits Events on the returned channel.

type Event

type Event struct {
	Type       EventType
	Delta      string         // for EventTextDelta
	ToolUse    *types.ToolUse // for EventToolUse
	StopReason string         // for EventDone
	Err        error          // for EventError
	Usage      *Usage         // optional, on EventDone
}

Event is a streamed token, tool call, or terminal signal from a provider.

type EventType

type EventType string
const (
	EventTextDelta     EventType = "text_delta"
	EventThinkingDelta EventType = "thinking_delta"
	EventToolUse       EventType = "tool_use"
	EventDone          EventType = "done"
	EventError         EventType = "error"
)

type GeminiConfig

type GeminiConfig struct {
	BaseURL    string
	APIKey     string
	HTTPClient *http.Client
}

GeminiConfig configures the native Gemini provider. BaseURL defaults to Google's generative-language v1beta endpoint; APIKey is appended as a query param (Google's recommended auth path for this API).

type GeminiProvider

type GeminiProvider struct {
	// contains filtered or unexported fields
}

GeminiProvider streams from Google's generative-language API via :streamGenerateContent?alt=sse.

func NewGemini

func NewGemini(cfg GeminiConfig) *GeminiProvider

NewGemini builds a provider. APIKey is read from GEMINI_API_KEY env when the config field is blank, mirroring how callers wire other providers.

func (*GeminiProvider) Name

func (p *GeminiProvider) Name() string

Name returns the static provider identifier.

func (*GeminiProvider) Stream

func (p *GeminiProvider) Stream(ctx context.Context, req Request) (<-chan Event, error)

Stream issues streamGenerateContent and emits Events on the returned channel. Caller must read until close; ctx cancellation closes the channel after an EventError. Gemini SSE: each line `data: <json>\n\n`; the model terminates by emitting `finishReason: "STOP"` (no separate `[DONE]` marker, but we handle one defensively).

type Model

type Model struct {
	ID            string
	Name          string
	ContextLength int
	Pricing       string
}

Model is a single entry in a provider's catalogue.

func FetchChatGPTModels

func FetchChatGPTModels(ctx context.Context, baseURL, accountIDHeader string) ([]Model, error)

FetchChatGPTModels GETs /backend-api/codex/models?client_version=. The server returns a plan-filtered model list once authed; without a high enough client_version it only surfaces a single legacy entry. We send 9.9.9 to opt into the full set the account is entitled to.

func ListModels

func ListModels(ctx context.Context, name string, cfg config.ProviderConfig) ([]Model, error)

ListModels returns the model catalogue for the given provider. For anthropic-wire providers or empty base URLs we return a curated hardcoded list — Anthropic's /models endpoint is gated and not all wire-compat servers expose one. Results are cached per-name with a 10-minute TTL.

type OpenAICompatConfig

type OpenAICompatConfig struct {
	// Name shows up in logs and Provider.Name(). Defaults to "openai-compat".
	Name string
	// BaseURL e.g. "https://openrouter.ai/api/v1". The "/chat/completions"
	// suffix is appended automatically.
	BaseURL string
	// EnvKey is the environment variable holding the bearer token. Optional —
	// when blank (local servers like Ollama), no Authorization header is sent.
	EnvKey string
	// HTTPClient is overridable for tests. Defaults to a long-timeout client.
	HTTPClient *http.Client
	// ExtraHeaders are merged into every request. Useful for OpenRouter's
	// HTTP-Referer + X-Title attribution headers.
	ExtraHeaders map[string]string
	// StallTimeout overrides the per-chunk SSE inactivity timer. 0 falls back
	// to the package default (streamStallTimeout). Negative disables the
	// watchdog (test-only).
	StallTimeout time.Duration
	// ChatTemplateKwargs flows into the MLX/vllm `chat_template_kwargs` body
	// field. Lets local servers flip Qwen3 template switches that change tool
	// emission shape. Nil = omit. Per-provider (TOML `chat_template_kwargs`).
	ChatTemplateKwargs map[string]any
	// ReportsCost opts the provider into a request flag asking for actual spend
	// in the usage block. Off by default; strict endpoints would reject the
	// extra field, so only set it for services that return real per-call cost.
	ReportsCost bool
}

OpenAICompatConfig configures a chat-completions-style provider. The same implementation covers OpenRouter, OpenAI, DeepSeek, Groq, Ollama, LM Studio, Together, Fireworks — any service that speaks OpenAI's wire format.

type OpenAICompatProvider

type OpenAICompatProvider struct {
	// contains filtered or unexported fields
}

OpenAICompatProvider implements Provider against an OpenAI-compatible API.

func NewOpenAICompat

func NewOpenAICompat(cfg OpenAICompatConfig) *OpenAICompatProvider

NewOpenAICompat builds a provider. Missing fields get sensible defaults.

func (*OpenAICompatProvider) Name

func (p *OpenAICompatProvider) Name() string

Name returns the configured display name.

func (*OpenAICompatProvider) Stream

func (p *OpenAICompatProvider) Stream(ctx context.Context, req Request) (<-chan Event, error)

Stream issues a chat completion and emits Events on the returned channel. Caller must read the channel until closed to avoid leaking the goroutine; ctx cancellation closes the channel after an EventError.

Retries the pre-stream phase (DNS/TCP/TLS failures, 408, 429, 5xx) with exp-backoff. Once a 2xx response lands and streamLoop starts emitting, no further retries — replaying would duplicate tokens.

type Provider

type Provider interface {
	Name() string
	Stream(ctx context.Context, req Request) (<-chan Event, error)
}

Provider streams a chat completion. The returned channel is closed when the turn ends (stop reason emitted) or the context is canceled.

type Request

type Request struct {
	Model       string
	System      string
	Messages    []types.Message
	Tools       []ToolSpec
	MaxTokens   int
	Temperature float64
	// TopP pins nucleus sampling. 0 = use provider default (omit on wire).
	// Set by the active profile (tiny pins 0.8 for 4-bit MoE).
	TopP float64
	// Stop is the optional list of stop sequences. textmode wraps tool calls
	// in `<bash>{…}</bash>`-style envelopes; passing the closing tag halts
	// decode immediately after one tool call, saving 50–300 tokens per turn.
	Stop   []string
	Stream bool
	// Thinking selects the extended-reasoning budget for providers that
	// support it. Off means omit the field entirely.
	Thinking Thinking
	// ChatTemplateKwargs overrides/merges with the provider's static
	// chat_template_kwargs for this request. Used to flip Qwen3's
	// enable_thinking switch per-turn from the effort level. Nil = no override.
	ChatTemplateKwargs map[string]any
}

Request is the agent-owned shape of a chat call. Adapters translate to the provider's wire format.

type TextModeOptions

type TextModeOptions struct {
	// ExtraHint is appended verbatim after the auto-generated tool block.
	// Useful for caveman-style brevity nudges.
	ExtraHint string
}

TextModeOptions tunes the wrapper. All fields are optional.

type TextModeProvider

type TextModeProvider struct {
	// contains filtered or unexported fields
}

TextModeProvider wraps an inner Provider to bypass the native tool_calls channel. Many small local models (llama3.1:8b, gemma3, phi3) silently ignore function-calling deltas but reliably emit inline XML when shown one example. The wrapper:

  • strips Request.Tools and injects a text instruction block describing each tool plus the `<tool>{...}</tool>` envelope,
  • buffers assistant text deltas,
  • scans the buffered text after the stream ends for tool-call tags, synthesizes EventToolUse for each, and emits the cleaned text.

Why a wrapper instead of a separate provider: every existing adapter (openai_compat, chatgpt, claude, gemini) gains XML-mode for free.

func NewTextMode

func NewTextMode(inner Provider, opts TextModeOptions) *TextModeProvider

NewTextMode wraps inner with the text/XML tool-call fallback.

func (*TextModeProvider) Name

func (p *TextModeProvider) Name() string

Name forwards to inner with a "+textmode" suffix so logs/UIs can tell.

func (*TextModeProvider) Stream

func (p *TextModeProvider) Stream(ctx context.Context, req Request) (<-chan Event, error)

Stream injects the text-tool instruction block, nils Tools, then runs the inner stream. Tool-call extraction happens at EventDone.

Side-LLM calls (mode classifier, recap, compact) pass req.Tools == nil. In that case skip injection entirely — pumping a `## Tools (text format)` block with an empty tool list into a classifier/recap prompt pollutes the instruction and pushes small models toward emitting spurious XML.

type Thinking

type Thinking string

Thinking enumerates the supported extended-reasoning levels. Adapters map these to provider-specific fields (Anthropic budget_tokens, OpenAI reasoning_effort).

const (
	// ThinkingAuto = "medium when model supports reasoning, off otherwise".
	// Resolve with ResolveThinking before sending to providers — wire layers
	// only understand off/low/medium/high.
	ThinkingAuto   Thinking = "auto"
	ThinkingOff    Thinking = "off"
	ThinkingLow    Thinking = "low"
	ThinkingMedium Thinking = "medium"
	ThinkingHigh   Thinking = "high"
	// ThinkingMax pushes the reasoning budget to the provider's practical
	// ceiling. Wire layers without a distinct "max" tier (OpenAI's
	// reasoning_effort) clamp to "high"; budget-token providers (Anthropic,
	// Gemini) get a much larger budget than High.
	ThinkingMax Thinking = "max"
)

func ParseThinking

func ParseThinking(s string) Thinking

ParseThinking accepts "auto"/"off"/"low"/"medium"/"high"/"max" (case insensitive) and returns the canonical Thinking value. Unknown strings return ThinkingOff.

func ResolveThinking

func ResolveThinking(t Thinking, modelID string) Thinking

ResolveThinking turns ThinkingAuto into ThinkingMedium for reasoning-capable models and ThinkingOff for everything else. Non-auto values pass through unchanged. Call before building a Request — provider adapters only see the resolved level, never the sentinel.

type ToolSpec

type ToolSpec struct {
	Name          string         `json:"name"`
	Description   string         `json:"description"`
	PromptSnippet string         `json:"-"`
	Schema        map[string]any `json:"schema"`
}

ToolSpec is the agent-side advertisement of a tool to the model.

Description goes to the provider's tool-spec (API) — full sentence is fine. PromptSnippet is the short prompt-manifest line (≤60 chars). When PromptSnippet is empty, the prompt manifest falls back to the first line of Description, truncated by the profile's ToolDescChars budget.

type Usage

type Usage struct {
	InputTokens  int
	OutputTokens int
	// CachedTokens is the cached prompt-token count, when the provider reports
	// it. Zero otherwise — most providers omit it.
	CachedTokens int
	// CostUSD is the provider-reported actual spend for this turn, when the
	// service returns it (some routed aggregators do). Zero when unreported;
	// callers then fall back to a static price-table estimate.
	CostUSD float64
}

Usage captures token accounting reported by the provider.

Directories

Path Synopsis
Package mockprov is a scripted, deterministic llm.Provider for tests.
Package mockprov is a scripted, deterministic llm.Provider for tests.
anthropic_messages.go covers Anthropic's native Messages API (POST /v1/messages, SSE streaming).
anthropic_messages.go covers Anthropic's native Messages API (POST /v1/messages, SSE streaming).

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL