llm

package
v0.6.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 8, 2026 License: MIT Imports: 21 Imported by: 0

Documentation

Index

Constants

View Source
const (
	// BackendChatCompletions is OpenAI's /chat/completions, what almost every
	// local server and gateway offers. It is the default.
	BackendChatCompletions = "chat_completions"
	// BackendResponses is OpenAI's /responses. Some hosted models are served
	// only there and answer /chat/completions with an error.
	BackendResponses = "responses"
)

Backend names the wire protocol the endpoint speaks.

View Source
const (
	// EffortFieldChatTemplateKwargs puts the effort in
	// `chat_template_kwargs.reasoning_effort`, which is how a chat template
	// running on a local llama.cpp or vLLM server reads it (the Qwen template
	// does). This is the default, because that is where the setting started.
	EffortFieldChatTemplateKwargs = "chat_template_kwargs"
	// EffortFieldReasoningEffort puts it in the top-level `reasoning_effort`
	// field, which is OpenAI's own spelling and what hosted gateways expect.
	// opencode.ai's /chat/completions wants this one and drops
	// chat_template_kwargs on the floor.
	EffortFieldReasoningEffort = "reasoning_effort"
)

EffortField names where a chat-completions endpoint reads the reasoning effort from. The two spellings are not interchangeable: an endpoint that wants one ignores the other silently, so the effort a person picked with :eN does nothing and nothing says so.

View Source
const (
	DefaultImageMaxSizeMB = 15
	DefaultImageMaxDim    = 1568
)

Defaults for image ingestion. A full-resolution screenshot costs thousands of tokens for no additional accuracy on code-adjacent images, so anything larger than the long edge is downscaled once, at ingest, and the normalized copy is what every later turn re-encodes.

View Source
const BackendMessages = "messages"

BackendMessages is the Anthropic Messages API (POST /v1/messages).

Variables

This section is empty.

Functions

func EstimateImageTokens

func EstimateImageTokens(width, height int) int

EstimateImageTokens approximates the tokens a provider charges for an image: a base cost plus a per-tile cost over 512px tiles. It is the rule OpenAI publishes; other providers differ, but only in the constants, so it is good enough for a client-side context guard.

func ExtractNonToolCallContent

func ExtractNonToolCallContent(content string) string

ExtractNonToolCallContent extracts the non-tool-call parts from content that contains XML-like tool call descriptions

func ImageTokensForMessages

func ImageTokensForMessages(msgs []Message) int

ImageTokensForMessages sums the estimate over every attached image.

func IsImagePath

func IsImagePath(path string) bool

IsImagePath reports whether path names an image by extension. It says nothing about whether the file exists or what it contains; PrepareImage sniffs the content rather than trusting this.

func IsWSL

func IsWSL() bool

IsWSL reports whether this process runs under Windows Subsystem for Linux, where the clipboard lives on the Windows side (see internal/tui).

func NormalizeImagePath

func NormalizeImagePath(path string) string

NormalizeImagePath cleans a raw image reference from the composer: quoted drag-drop paths, file:// URLs, and — on WSL — Windows paths from Windows Terminal. It is pure string work plus wslpath when one is needed.

func NormalizeToolCallTypes

func NormalizeToolCallTypes(msg *Message)

NormalizeToolCallTypes ensures all tool calls have type: "function"

func ValidEffortField

func ValidEffortField(f string) bool

ValidEffortField reports whether f names a known effort field (or is empty, which means the chat-template default).

Types

type ChatRequest

type ChatRequest struct {
	Model       string     `json:"model"`
	Messages    []Message  `json:"messages"`
	Temperature float32    `json:"temperature,omitempty"`
	MaxTokens   int        `json:"max_tokens,omitempty"`
	Tools       []ToolSpec `json:"tools,omitempty"`
	ToolChoice  string     `json:"tool_choice,omitempty"`
	Stream      bool       `json:"stream,omitempty"`
	// ChatTemplateKwargs passes extra args to the server's chat template (llama.cpp /
	// vLLM style), e.g. {"enable_thinking": false} to suppress reasoning for a call.
	// Omitted from the body when nil; ignored by templates that don't use the key.
	ChatTemplateKwargs map[string]any `json:"chat_template_kwargs,omitempty"`
	// ReasoningEffort is OpenAI's own top-level effort field, which hosted
	// gateways read instead of ChatTemplateKwargs. Which of the two a client
	// fills is WithEffortField; see the EffortField constants.
	ReasoningEffort string `json:"reasoning_effort,omitempty"`
}

type ChatResponse

type ChatResponse struct {
	ID      string   `json:"id"`
	Model   string   `json:"model"`
	Choices []Choice `json:"choices"`
	Usage   Usage    `json:"usage"`
}

type Choice

type Choice struct {
	Index        int          `json:"index"`
	Message      Message      `json:"message"`
	FinishReason string       `json:"finish_reason"`
	Error        *ChoiceError `json:"error,omitempty"`
}

Choice is one candidate answer. Every endpoint kvit-coder talks to returns exactly one, so the loop only ever reads Choices[0].

type ChoiceError

type ChoiceError struct {
	Message string `json:"message"`
	Code    int    `json:"code"`
}

ChoiceError represents an error returned in a choice (e.g., upstream provider errors)

type Client

type Client struct {
	// contains filtered or unexported fields
}

func NewClient

func NewClient(baseURL, apiKey string, opts ...Option) *Client

func (*Client) Chat

func (c *Client) Chat(ctx context.Context, req ChatRequest) (*ChatResponse, error)

Chat sends one completion request and returns the model's answer. Which wire protocol it speaks is decided by the backend the client was built with: OpenAI chat completions by default, or the Responses API when the endpoint only serves the model there.

func (*Client) GetGenerationStats

func (c *Client) GetGenerationStats(ctx context.Context, generationID string) (*GenerationStats, error)

GetGenerationStats queries the generation statistics for a given generation ID

type GenerationStats

type GenerationStats struct {
	Data struct {
		ID                     string  `json:"id"`
		CreatedAt              string  `json:"created_at"`
		Model                  string  `json:"model"`
		Streamed               bool    `json:"streamed"`
		FinishReason           string  `json:"finish_reason"`
		TokensPrompt           int     `json:"tokens_prompt"`
		TokensCompletion       int     `json:"tokens_completion"`
		NativeTokensPrompt     int     `json:"native_tokens_prompt"`
		NativeTokensCompletion int     `json:"native_tokens_completion"`
		NativeTokensCached     int     `json:"native_tokens_cached"`
		TotalCost              float64 `json:"total_cost"`
		CacheDiscount          float64 `json:"cache_discount"`
		ProviderName           string  `json:"provider_name"`
		InternalProvider       string  `json:"internal_provider"`
		// Timing fields (in milliseconds)
		Latency        float64 `json:"latency"`         // Time to process (native_tokens_prompt - native_tokens_cached)
		GenerationTime float64 `json:"generation_time"` // Time to generate native_tokens_completion
	} `json:"data"`
}

GenerationStats contains detailed generation statistics from the LLM server

type ImagePart

type ImagePart struct {
	Path      string `json:"path,omitempty"`
	MediaType string `json:"media_type,omitempty"`
	Width     int    `json:"width,omitempty"`
	Height    int    `json:"height,omitempty"`
	Size      int64  `json:"size,omitempty"`
	Data      []byte `json:"-"`
}

ImagePart is one image attached to a message. The pixels live on disk — in the session's tmp/ once normalized — and Path points at that copy; Data carries the bytes in memory only (json:"-"), filled by HydrateImages before a request is built. History therefore stores references, never base64.

func PrepareImage

func PrepareImage(srcPath, destDir string, maxSizeMB, maxDim int) (ImagePart, error)

PrepareImage validates srcPath as an image, normalizes it into destDir, and returns the attachment pointing at the normalized copy with bytes loaded. Oversized images are downscaled (long edge to maxDim) and re-encoded as JPEG; anything else is stored byte-identical. Non-positive caps select the defaults.

func (ImagePart) DataURL

func (p ImagePart) DataURL() string

DataURL renders the in-memory bytes as a data: URL for the wire.

func (ImagePart) Name

func (p ImagePart) Name() string

Name is the file name for display, never the full path.

func (ImagePart) Summary

func (p ImagePart) Summary() string

Summary is the one-line redaction used in logs, transcripts and echoes: dimensions and size, never pixels.

type Message

type Message struct {
	Role             MessageRole `json:"role"`
	Content          string      `json:"content,omitempty"`
	ReasoningContent string      `json:"reasoning_content,omitempty"` // For models that return thinking/reasoning
	Name             string      `json:"name,omitempty"`
	ToolCalls        []ToolCall  `json:"tool_calls,omitempty"`
	ToolCallID       string      `json:"tool_call_id,omitempty"` // For tool role messages

	// ReasoningBlocks holds the opaque thinking blocks a Responses-API model
	// returned with this assistant turn. The model expects them back verbatim
	// on the next request of the same tool loop, so they ride along in the
	// history but are never sent on the wire by the chat-completions path.
	ReasoningBlocks []ReasoningBlock `json:"-"`
	// ToolCallItemIDs maps a tool call ID to the Responses-API item ID the
	// model gave that call, so a replayed function_call keeps its identity.
	ToolCallItemIDs map[string]string `json:"-"`
	// Images are pictures attached to the message. The pixels live on disk
	// (the session's tmp/ once normalized) and Path points at the copy;
	// bytes are loaded into Data by HydrateImages before a request is built,
	// so history stores references, never base64. Images ride on user
	// messages; other roles ignore them.
	Images []ImagePart `json:"images,omitempty"`
}

func HydrateImages

func HydrateImages(msgs []Message) []Message

HydrateImages loads every attachment's bytes from disk, returning a copy. A file that is gone (tmp/ cleaned out from under the session, a hand-edited history) degrades to a text notice on that message rather than failing the turn. Messages without attachments are returned untouched.

func PreventConsecutiveAssistant

func PreventConsecutiveAssistant(messages []Message) ([]Message, bool)

PreventConsecutiveAssistant checks if the last message is an assistant message and removes it if so, returning true if a message was removed.

func (*Message) UnmarshalJSON

func (m *Message) UnmarshalJSON(data []byte) error

UnmarshalJSON accepts content as the string every writer stores, or as the array of typed parts some endpoints return, so a multimodal answer decodes without error. Image parts on the wire carry no bytes back, so only their text survives; attachments kvit-coder sent are already in Images.

type MessageRole

type MessageRole string
const (
	RoleSystem    MessageRole = "system"
	RoleUser      MessageRole = "user"
	RoleAssistant MessageRole = "assistant"
	RoleTool      MessageRole = "tool"
)

type Option

type Option func(*Client)

Option adjusts a Client at construction time.

func WithAPIKeyEnv

func WithAPIKeyEnv(name string) Option

WithAPIKeyEnv records which environment variable the key came from, so a 401 can name it. An endpoint that needs no key leaves this empty.

func WithBackend

func WithBackend(backend string) Option

WithBackend selects the wire protocol. An empty or unknown value leaves the client on chat completions.

func WithEffortField

func WithEffortField(field string) Option

WithEffortField picks which field of a chat-completions request carries the reasoning effort: EffortFieldChatTemplateKwargs (the default, for a model whose chat template reads it) or EffortFieldReasoningEffort (OpenAI's own top-level field, which hosted gateways want). An empty or unknown value leaves the default. The Responses backend ignores this.

func WithHeaders

func WithHeaders(headers map[string]string) Option

WithHeaders adds request headers sent with every call, on top of Content-Type and Authorization. Use it for endpoints that demand something extra, such as a routing or session header.

func WithMaxRetries

func WithMaxRetries(n int) Option

WithMaxRetries caps how many times a failed request is tried again, for a caller that would rather fall back than wait. Zero means one attempt and no retry; a negative value leaves the default of defaultMaxRetries.

The retry ladder waits 1s, 2s, 4s, 8s, 16s and so on between attempts, and an endpoint that refuses instantly — a 502 from a local server with no model loaded, say — spends all of that waiting. A request whose answer is optional should not: it holds up whatever is waiting on it and there is nothing on screen to say why.

func WithReasoningEffort

func WithReasoningEffort(effort string) Option

WithReasoningEffort sets how much thinking a reasoning model should do. The Responses backend sends it as `reasoning.effort`; the chat-completions backend sends it wherever WithEffortField points. Empty means "server default" on both.

func WithReasoningSummary

func WithReasoningSummary(summary string) Option

WithReasoningSummary asks a reasoning model for readable text describing its thinking, which is the only part of its reasoning anyone can read: the rest comes back encrypted, to be replayed rather than understood. Only the Responses backend sends it; the accepted values are the provider's (commonly auto, concise, detailed). Without it merge_thinking has nothing to merge.

func WithRequestAuthorizer added in v0.3.0

func WithRequestAuthorizer(auth RequestAuthorizer) Option

WithRequestAuthorizer installs an authorizer. Nil leaves requests unchanged.

func WithRetryNotice

func WithRetryNotice(fn func(attempt, maxAttempts int, delay time.Duration, reason error)) Option

WithRetryNotice installs a callback told about each failed attempt, so a caller can say so rather than leaving a stalled turn looking like a slow one.

func WithTimeout

func WithTimeout(d time.Duration) Option

WithTimeout bounds one HTTP request end to end, overriding the default of ten minutes. A non-positive value leaves the default in place.

type ReasoningBlock

type ReasoningBlock struct {
	ID               string `json:"id,omitempty"`
	EncryptedContent string `json:"encrypted_content,omitempty"`
	Summary          []any  `json:"summary,omitempty"`
}

ReasoningBlock is one "reasoning" item from a Responses-API answer. The content is encrypted by the provider: kvit-coder cannot read it, it only stores it and hands it back.

type RequestAuthorizer added in v0.3.0

type RequestAuthorizer interface {
	Authorize(ctx context.Context, req *http.Request) error
	Invalidate()
}

RequestAuthorizer adjusts a request immediately before it is sent. Authorize returning an error fails the request without the retry ladder. Invalidate drops a cached credential so the next Authorize fetches a new one; postJSON calls it once after a 401.

type ResponseNormalizer

type ResponseNormalizer struct {
	// contains filtered or unexported fields
}

ResponseNormalizer handles normalization of LLM responses

func NewResponseNormalizer

func NewResponseNormalizer(extractor ToolCallExtractor, mergeThinking bool) *ResponseNormalizer

NewResponseNormalizer creates a new response normalizer

func (*ResponseNormalizer) NormalizeResponse

func (n *ResponseNormalizer) NormalizeResponse(msg *Message) bool

NormalizeResponse applies all middleware transformations to an LLM response. Returns true if tool calls were extracted (requires updating finish reason).

type StatusError added in v0.6.2

type StatusError struct {
	Code int
	Body []byte
}

StatusError is an answer with an HTTP status other than 200. Its text is the one this client has always printed; the type exists so that a caller that has to act on the code, such as kvit-coder-ui telling a refused key from a protocol the endpoint does not serve, can find it with errors.As.

func (*StatusError) Error added in v0.6.2

func (e *StatusError) Error() string

type ToolCall

type ToolCall struct {
	ID       string           `json:"id"`
	Type     string           `json:"type"`
	Function ToolCallFunction `json:"function"`
}

type ToolCallExtractor

type ToolCallExtractor interface {
	ExtractToolCallsFromText(content string) []ToolCall
}

ToolCallExtractor is an interface for extracting tool calls from text

type ToolCallFunction

type ToolCallFunction struct {
	Name      string `json:"name"`
	Arguments string `json:"arguments"` // JSON string
}

ToolCallFunction is the function half of a tool call: which tool, and the arguments as the JSON string the model produced.

type ToolSpec

type ToolSpec struct {
	Type     string           `json:"type"` // always "function"
	Function ToolSpecFunction `json:"function"`
}

type ToolSpecFunction

type ToolSpecFunction struct {
	Name        string         `json:"name"`
	Description string         `json:"description"`
	Parameters  map[string]any `json:"parameters"`
}

ToolSpecFunction is the schema half of a tool spec, as the API expects it.

type Usage

type Usage struct {
	PromptTokens     int `json:"prompt_tokens"`
	CompletionTokens int `json:"completion_tokens"`
	TotalTokens      int `json:"total_tokens"`
}

Usage is the token accounting the endpoint reports for one request.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL