inference

package module
v0.26.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 12, 2026 License: MIT Imports: 15 Imported by: 0

README

LLM Inference for Go

lint test

inference-go is a normalized Go interface for language-model inference across vendor APIs, OpenAI/Anthropic/Google-compatible services, routers, and local runtimes.

It gives applications one request and response model while preserving meaningful provider and model differences through capability profiles and model presets.

Installation

Requires Go 1.26 or newer.

Install with go get github.com/flexigpt/inference-go.

What you get

  • One completion API through ProviderSetAPI
  • Anthropic Messages, OpenAI Responses, OpenAI Chat Completions, and Google Generate Content API format adapters
  • Provider and model presets through modelpreset
  • Text, image, and file input where supported by the selected model
  • Text and thinking streaming where the upstream API exposes it
  • Reasoning controls with model-specific validation
  • Structured output and JSON Schema where supported
  • Function, custom, and web-search tool abstractions
  • Cache-control normalization where supported
  • Usage normalization for input, output, cached, and reasoning tokens where exposed
  • Capability-driven warnings when a safe setting is removed
  • Early errors for unsupported contract-like settings such as an unavailable output format
  • Model-level capability overrides for gateways, routers, and local runtimes
  • HTTP debugging through CompletionDebugger and debugclient

Supported provider presets

The preset catalog currently includes the following providers.

Provider preset Adapter family Typical use
Anthropic Anthropic Messages Claude models and Anthropic-native thinking/tool flows
DeepSeek OpenAI Responses DeepSeek hosted models
Google Gemini Google Generate Content Gemini models, native thinking, grounding, and files
Hugging Face Router OpenAI Chat Completions Routed hosted models with backend-specific identities
LocalAI OpenAI Responses Local OpenAI-compatible serving
LM Studio OpenAI Responses Local desktop/server inference
llama.cpp OpenAI Chat Completions Local OpenAI-compatible serving
Meta OpenAI Responses Meta-hosted Muse Spark models
MiniMax OpenAI Responses MiniMax hosted models
Mistral OpenAI Chat Completions Mistral-hosted models
Moonshot Anthropic Messages Kimi and Moonshot models
Ollama Anthropic Messages Local Anthropic-compatible serving
OpenAI Chat Completions OpenAI Chat Completions OpenAI Chat models and compatible endpoints
OpenAI Responses OpenAI Responses OpenAI Responses models
OpenRouter OpenAI Responses Routed models with model-specific overrides
QwenCloud OpenAI Responses Qwen/DashScope compatible-mode models
SGLang OpenAI Responses Self-hosted OpenAI-compatible serving
vLLM OpenAI Responses Self-hosted OpenAI-compatible serving
xAI OpenAI Responses Grok and xAI-hosted models
Xiaomi OpenAI Responses MiMo hosted models
Z.AI OpenAI Chat Completions Z.AI pay-as-you-go models
Z.AI Coding Plan OpenAI Responses Z.AI Coding Plan models and credentials

A provider adapter is not a promise that every model supports every adapter feature. The selected model preset and its effective capability resolver determine the actual request surface.

See modelpreset/README.md for catalog guidance and the current provider/model grouping.

Feature overview

Feature What inference-go provides
Text input and output Supported across all adapters
Image input Supported by selected adapters and models
File input Supported by selected adapters and models
Text streaming Supported where the upstream provider streams text
Thinking streaming Supported by Anthropic, OpenAI Responses, and Google Generate Content where exposed upstream
Reasoning controls Capability-aware validation and model-specific restrictions
JSON Schema output Supported by selected adapters and models
Function tools Supported across the main adapters
Custom tools Supported where the selected adapter/model can represent them
Web search Supported through provider-native web-search or grounding features where available
Cache control Supported where the provider/model exposes compatible cache behavior
Usage accounting Normalized where upstream usage data is available
HTTP debugging Supported through CompletionDebugger

What is not yet a portable feature

The library intentionally does not yet provide a general cross-provider abstraction for:

  • Audio input or output
  • Video input or output
  • Image generation or image output
  • Provider-native stored conversations
  • Previous-response identifiers
  • Background jobs
  • Prompt resources
  • Uploaded-file lifecycle management
  • Arbitrary provider-specific request parameters

These features may be available from individual vendors. They are not yet part of the stable normalized surface.

Start with a preset

For most applications, start with modelpreset.

The typical workflow is:

  1. Select a provider preset.
  2. Select a model preset for that provider.
  3. Add the provider connection through AddProviderFromPreset.
  4. Set the API key through SetProviderAPIKey.
  5. Build a model-specific resolver with NewPresetCapabilityResolver.
  6. Pass that resolver through FetchCompletionOptions.

The resolver matters because providers, routers, and local servers often host models with different reasoning, tool, modality, output, or cache support.

See the executable setup example in internal/integration/example_preset_catalog_test.go.

Working examples

Use the integration examples as the current source of truth for end-to-end flows.

Scenario Example
Preset lookup, provider registration, and resolver setup example_preset_catalog_test.go
Anthropic basic completion example_anthropic_basic_test.go
Anthropic tools, thinking, and streaming example_anthropic_tools_streaming_test.go
OpenAI Responses basic completion example_openai_responses_basic_test.go
OpenAI Responses tools, files, images, and streaming example_openai_responses_tools_attachments_test.go
Meta Muse Spark basic completion example_meta_muse_spark_test.go
Meta Muse Spark multi-parameter request example_meta_muse_spark_test.go
OpenAI Chat Completions basic completion example_openai_chat_basic_test.go
OpenAI Chat tools, JSON Schema, and streaming example_openai_chat_tools_websearch_stream_test.go
Google Generate Content basic completion example_google_genai_basic_test.go
Google function-tool round trip example_google_genai_tools_roundtrip_test.go
Google web search, thinking, and streaming example_google_genai_websearch_streaming_test.go
Capability overrides example_capability_override_test.go

The live examples read credentials from environment variables. They remain useful implementation references even when credentials are unavailable.

Capability-aware behavior

Each completion is validated and normalized against effective model capabilities.

Effective capabilities are usually derived from:

  1. the adapter's base capability profile
  2. the provider preset override
  3. the model preset override
  4. an optional caller override

The normalizer can:

  • Reject a request when it asks for something the selected model cannot safely provide
  • Remove safe optional settings and return warnings
  • Narrow requests to model-specific limits
  • Sanitize provider-specific reasoning history before it is replayed
  • Adapt tool-output transport to the selected model

For routers and local runtimes, add a caller override only when the deployed server or model differs from catalog metadata, and test that exact deployment before relying on advanced features.

Applications own conversation history. Preserve tool-call IDs and provider-native reasoning data only for compatible continuations; prefer ordinary user and assistant text when moving between provider families.

Detailed guides

Topic Guide
Provider and model preset catalog modelpreset/README.md
Anthropic Messages behavior internal/anthropicsdk/README.md
OpenAI Responses behavior internal/openairesponsessdk/README.md
OpenAI Chat Completions behavior internal/openaichatsdk/README.md
Google Generate Content behavior internal/googlegeneratecontentsdk/README.md

Development

  • Formatting/linting uses the repository configuration in .golangci.yml
  • Useful scripts are available in taskfile.yml
  • PRs are welcome
    • keep the public surface small and provider-neutral
    • avoid leaking provider SDK types into package inference or spec

License

Copyright (c) 2026 - Present - Pankaj Pipada

All source code in this repository, unless otherwise noted, is licensed under the MIT License. See LICENSE for details.

Documentation

Overview

Package inference provides a single, normalized interface for getting language model completions from multiple providers.

The main entry point is ProviderSetAPI, which lets you:

  • register one or more providers (Anthropic, OpenAI Chat Completions, OpenAI Responses, ...),
  • configure and rotate API keys,
  • send normalized completion requests and receive normalized outputs,
  • optionally stream partial text / reasoning and capture HTTP‑level debug information.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type AddProviderConfig 

type AddProviderConfig struct {
	SDKType                  spec.ProviderSDKType `json:"sdkType"`
	Origin                   string               `json:"origin"`
	ChatCompletionPathPrefix string               `json:"chatCompletionPathPrefix"`
	APIKeyHeaderKey          string               `json:"apiKeyHeaderKey"`
	DefaultHeaders           map[string]string    `json:"defaultHeaders"`
}

type DebugClientBuilder 

type DebugClientBuilder func(p spec.ProviderParam) spec.CompletionDebugger

DebugClientBuilder constructs a CompletionDebugger for a given provider. A nil builder or a nil returned debugger disable debugging for that provider.

type ProviderSetAPI 

type ProviderSetAPI struct {
	// contains filtered or unexported fields
}

func NewProviderSetAPI 

func NewProviderSetAPI(
	opts ...ProviderSetOption,
) (*ProviderSetAPI, error)

NewProviderSetAPI creates a new ProviderSet and installs the process-wide logger used by this SDK. The logger is chosen via WithLoggerBuilder; if no builder is provided or it returns nil, a no-op logger is used.

func (*ProviderSetAPI) AddProvider 

func (ps *ProviderSetAPI) AddProvider(
	ctx context.Context,
	provider spec.ProviderName,
	config *AddProviderConfig,
) (spec.ProviderParam, error)

func (*ProviderSetAPI) AddProviderFromPreset added in v0.20.0 

func (ps *ProviderSetAPI) AddProviderFromPreset(
	ctx context.Context,
	provider spec.ProviderName,
	preset modelpreset.ProviderPreset,
) (spec.ProviderParam, error)

func (*ProviderSetAPI) DeleteProvider 

func (ps *ProviderSetAPI) DeleteProvider(
	ctx context.Context,
	provider spec.ProviderName,
) error

func (*ProviderSetAPI) FetchCompletion 

func (ps *ProviderSetAPI) FetchCompletion(
	ctx context.Context,
	provider spec.ProviderName,
	fetchCompletionRequest *spec.FetchCompletionRequest,
	opts *spec.FetchCompletionOptions,
) (*spec.FetchCompletionResponse, error)

FetchCompletion processes a completion request for a given provider.

func (*ProviderSetAPI) GetProviderCapability added in v0.9.1 

func (ps *ProviderSetAPI) GetProviderCapability(
	ctx context.Context,
	provider spec.ProviderName,
) (spec.ModelCapabilities, error)

func (*ProviderSetAPI) NewPresetCapabilityResolver added in v0.20.0 

func (ps *ProviderSetAPI) NewPresetCapabilityResolver(
	ctx context.Context,
	runtimeProvider spec.ProviderName,
	providerPreset modelpreset.ProviderPreset,
	modelPreset modelpreset.ModelPreset,
	completionKey string,
) (spec.ModelCapabilityResolver, error)

func (*ProviderSetAPI) SetProviderAPIKey 

func (ps *ProviderSetAPI) SetProviderAPIKey(
	ctx context.Context,
	provider spec.ProviderName,
	apiKey string,
) error

SetProviderAPIKey sets the key for a given provider.

type ProviderSetOption 

type ProviderSetOption func(*ProviderSetAPI)

ProviderSetOption configures optional behavior for ProviderSetAPI.

func WithDebugClientBuilder 

func WithDebugClientBuilder(builder DebugClientBuilder) ProviderSetOption

WithDebugClientBuilder configures a CompletionDebugger factory. The builder is invoked once per provider when it is added. Returning nil disables debugging for that provider.

func WithLogger 

func WithLogger(logger *slog.Logger) ProviderSetOption

type SetProviderAPIKeyRequestBody 

type SetProviderAPIKeyRequestBody struct {
	APIKey string `json:"apiKey" required:"true"`
}

type SetProviderAPIKeyResponse 

type SetProviderAPIKeyResponse struct{}

Source Files

View all Source files

Directories

Path Synopsis
internal
anthropicsdk
googlegeneratecontentsdk
integration
Package integration contains example-style tests that demonstrate how to call the inference-go library against real providers.
 Package integration contains example-style tests that demonstrate how to call the inference-go library against real providers.
logutil
openaichatsdk
openairesponsessdk
sdkutil

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.