llm

package module
v0.14.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 25, 2026 License: Apache-2.0 Imports: 7 Imported by: 0

Documentation

Overview

Package llm is the batteries-included provider SDK layered on the neutral github.com/looprig/inference model-call contract. It owns the provider POLICY that inference deliberately does not carry: the known-provider registry, the provider/API-format truth table, provider auth requirements, provider default endpoint behavior, and the fail-closed model-validation preset. It also hosts the self-contained provider-security machinery (aci, e2e, tee) and the provider-specific SigV4 authenticator (llm/auth). inference never imports llm.

Index

Constants

View Source
const APIFormatBedrockConverse = model.APIFormatBedrockConverse

APIFormatBedrockConverse is retained as a provider-module alias for the neutral Bedrock Converse wire-dialect label owned by inference/model.

View Source
const AuthGCP auth.AuthKind = "gcp"

AuthGCP classifies Google Cloud application credentials or an exchanged Vertex access token. Vertex construction is direct because auto.New cannot safely discover or refresh ADC credentials from its APIKey input.

View Source
const AuthOAuth auth.AuthKind = "oauth"

AuthOAuth classifies a provider credential obtained through an OAuth flow. The auto factory receives only an APIKey-shaped secret and therefore treats OAuth providers as direct-construction providers until a caller supplies the exchanged access token explicitly.

View Source
const AuthServiceKey auth.AuthKind = "service_key"

AuthServiceKey classifies a provider service-key document that contains more than one credential or endpoint field.

View Source
const AuthSigV4 auth.AuthKind = "sigv4"

AuthSigV4 classifies the AWS Signature Version 4 credential a Bedrock provider requires. It is an auth.AuthKind value defined here rather than in inference: inference owns only the generic credential kinds (AuthNone, AuthAPIKey), while provider-specific request-signing schemes are llm policy.

View Source
const AuthToken auth.AuthKind = "token"

AuthToken classifies a bearer/PAT credential whose provider also needs a separate account or tenant identifier.

Variables

This section is empty.

Functions

func ValidateModel

func ValidateModel(m model.Model) error

ValidateModel is the fail-closed provider-policy preset that layers the known-provider truth table on top of model.Model's structural validation. model.Model.Validate is deliberately provider-policy-free (it only checks a non-empty Name and a syntactically safe non-empty BaseURL); this preset adds the checks inference dropped:

  • Provider must be a known backend (RequiredAuth errors on an unclassified one).
  • Provider must speak the model's APIFormat (fail-closed: an unsupported pair is rejected).
  • An empty BaseURL is only acceptable when the provider resolves a canonical or region-routed endpoint (allowsEmptyBaseURL); otherwise an explicit base is required.

It reproduces the pre-split harness Model.Validate behavior, returning a *model.ValidationError on the first rule violated. OriginCustom models validate identically to catalog rows; the lower trust in a custom model's Caps is a downstream gating concern, not this preset's.

Types

type AttestationError

type AttestationError struct {
	Reason string
	Err    error
}

AttestationError is a TEE attestation failure. Fail-closed: a request must never be sent to the provider when this error is returned. Err may be nil when the failure has no underlying cause to chain.

It lives in llm, not inference: attestation is provider-security policy (Phala/Chutes confidential inference), which the neutral model-call layer deliberately does not carry.

func (*AttestationError) Error

func (e *AttestationError) Error() string

func (*AttestationError) Unwrap

func (e *AttestationError) Unwrap() error

type AuthBinding

type AuthBinding struct {
	Provider  string
	Transport string
	Scheme    credentials.Scheme
	Usage     credentials.UsageClass
	Issuer    string
	Audience  string
}

AuthBinding is the complete authority identity accepted by one provider transport. Scheme and usage are intentionally not enough to identify a credential: provider, transport, issuer, and audience are part of the binding as well.

func (AuthBinding) Descriptor

func (b AuthBinding) Descriptor() (credentials.Descriptor, error)

Descriptor converts the policy binding into the credential descriptor used by a source and its leases. Label is presentation metadata and is therefore deliberately absent from an AuthBinding.

func (AuthBinding) Valid

func (b AuthBinding) Valid() bool

Valid reports whether the complete binding can be represented by a safe credential descriptor.

type AuthPolicy

type AuthPolicy struct {
	Accepted []AuthBinding
}

AuthPolicy is the immutable set of complete credential identities accepted by one provider client. A policy with no accepted bindings is invalid.

func AuthPolicyForModel

func AuthPolicyForModel(selected model.Model) (AuthPolicy, error)

AuthPolicyForModel is the one-way compatibility bridge from the legacy Provider.RequiredAuth registry to an exact credentials policy. It is kept in llm because inference must remain unaware of provider identity and policy.

func (AuthPolicy) Accepts

func (p AuthPolicy) Accepts(descriptor credentials.Descriptor) bool

Accepts reports whether descriptor has exactly one of the policy's accepted six-field identities. Descriptor.Label is presentation-only and is ignored.

func (AuthPolicy) Match

func (p AuthPolicy) Match(descriptor credentials.Descriptor) error

Match validates descriptor against the policy and returns a safe typed mismatch error when it is not accepted.

func (AuthPolicy) Validate

func (p AuthPolicy) Validate() error

Validate checks every accepted identity and rejects duplicate bindings. The input slice is never mutated; callers should treat a policy as immutable once it has been used to construct a client.

type AuthPolicyMismatchError

type AuthPolicyMismatchError struct{ Reason string }

AuthPolicyMismatchError reports a source or lease identity that is not authorized for the bound provider transport.

func (*AuthPolicyMismatchError) Error

func (e *AuthPolicyMismatchError) Error() string

type AuthRequiredError

type AuthRequiredError struct {
	Provider Provider
	Kind     auth.AuthKind
}

AuthRequiredError is returned by a provider factory when a provider that requires credentials is given none. Fail-closed. Carries no secret. Provider is the llm provider-policy label; Kind is the inference credential kind (including AuthSigV4).

func (*AuthRequiredError) Error

func (e *AuthRequiredError) Error() string

type CounterConstructor

type CounterConstructor string

CounterConstructor names a provider constructor without making it executable. It is a directive safe to expose in logs and user-facing configuration errors.

const (
	CounterConstructorBedrock CounterConstructor = "bedrock.NewCounter"
)

type CounterDirectConstructionError

type CounterDirectConstructionError struct {
	Provider Provider
	Reason   CounterDirectConstructionReason
	Use      CounterConstructor
}

CounterDirectConstructionError directs callers to an exact provider counter constructor whose required inputs cannot be supplied by auto.NewCounter.

func (*CounterDirectConstructionError) Error

type CounterDirectConstructionReason

type CounterDirectConstructionReason string

CounterDirectConstructionReason classifies why auto.NewCounter cannot build an exact counter from its (Model, APIKey) inputs.

const (
	// CounterDirectConstructionNeedsSigV4 means construction requires AWS SigV4
	// credentials and a region, neither of which can be represented by APIKey.
	CounterDirectConstructionNeedsSigV4 CounterDirectConstructionReason = "requires AWS SigV4 credentials and region"
)

type CounterSupportError

type CounterSupportError struct {
	Provider  Provider
	Reason    CounterSupportReason
	APIFormat model.APIFormat
}

CounterSupportError reports that a known provider has no exact context counter in llm for the selected dialect. All fields are secret-free and inspectable with errors.As.

func (*CounterSupportError) Error

func (e *CounterSupportError) Error() string

type CounterSupportReason

type CounterSupportReason string

CounterSupportReason classifies why an exact provider context counter is not available through the llm module. It never contains provider-controlled text.

const (
	// CounterSupportExactUnavailable means the provider exposes no exact counter
	// that llm can construct. Consumers may explicitly inject a local estimator;
	// this error never substitutes one.
	CounterSupportExactUnavailable CounterSupportReason = "exact provider context counter unavailable"
	// CounterSupportAPIFormatUnavailable means the provider has an exact counter,
	// but it cannot encode the model's validated API dialect.
	CounterSupportAPIFormatUnavailable CounterSupportReason = "exact provider context counter unavailable for API format"
)

type InvalidAuthPolicyError

type InvalidAuthPolicyError struct {
	Reason string
	Err    error
}

InvalidAuthPolicyError reports malformed provider policy metadata. It carries no credential or provider response data.

func (*InvalidAuthPolicyError) Error

func (e *InvalidAuthPolicyError) Error() string

func (*InvalidAuthPolicyError) Unwrap

func (e *InvalidAuthPolicyError) Unwrap() error

type Provider

type Provider string

Provider names the concrete backend an llm/auto factory dispatches on. Unknown values are rejected by ValidateModel; a provider constructor additionally enforces each provider's auth requirement. It is the provider-policy analogue of the opaque model.ProviderName label: inference carries no provider constants, no auth requirements, and no default endpoints — those live here.

const (
	ProviderLMStudio               Provider = "lmstudio"
	ProviderPhala                  Provider = "phala"
	ProviderChutes                 Provider = "chutes"
	ProviderOpenRouter             Provider = "openrouter"
	ProviderOpenAI                 Provider = "openai"
	ProviderAzure                  Provider = "azure"
	ProviderAzureCognitiveServices Provider = "azure-cognitive-services"
	ProviderAnthropic              Provider = "anthropic"
	ProviderXAI                    Provider = "xai"
	ProviderBedrock                Provider = "bedrock"
	Provider302AI                  Provider = "302ai"
	ProviderAtomicChat             Provider = "atomic-chat"
	ProviderBaseten                Provider = "baseten"
	ProviderCerebras               Provider = "cerebras"
	ProviderCloudflareAIGateway    Provider = "cloudflare-ai-gateway"
	ProviderCloudflareWorkersAI    Provider = "cloudflare-workers-ai"
	ProviderCortecs                Provider = "cortecs"
	ProviderDeepSeek               Provider = "deepseek"
	ProviderDeepInfra              Provider = "deepinfra"
	ProviderDigitalOcean           Provider = "digitalocean"
	ProviderFrogBot                Provider = "frogbot"
	ProviderFireworks              Provider = "fireworks-ai"
	ProviderGitLab                 Provider = "gitlab"
	ProviderGitHubCopilot          Provider = "github-copilot"
	ProviderGMICloud               Provider = "gmicloud"
	ProviderGoogleVertex           Provider = "google-vertex"
	ProviderGoogleVertexAnthropic  Provider = "google-vertex-anthropic"
	ProviderGroq                   Provider = "groq"
	ProviderHuggingFace            Provider = "huggingface"
	ProviderHelicone               Provider = "helicone"
	ProviderLlama                  Provider = "llama"
	ProviderLlamaCPP               Provider = "llama.cpp"
	ProviderIONet                  Provider = "io-net"
	ProviderMoonshot               Provider = "moonshotai"
	ProviderMiniMax                Provider = "minimax"
	ProviderNVIDIA                 Provider = "nvidia"
	ProviderNebius                 Provider = "nebius"
	ProviderOllama                 Provider = "ollama"
	ProviderOllamaCloud            Provider = "ollama-cloud"
	ProviderOpenCode               Provider = "opencode"
	ProviderOpenCodeGo             Provider = "opencode-go"
	ProviderLLMGateway             Provider = "llmgateway"
	ProviderSAP                    Provider = "sap-ai-core"
	ProviderSTACKIT                Provider = "stackit"
	ProviderOVHCloud               Provider = "ovhcloud"
	ProviderScaleway               Provider = "scaleway"
	ProviderSnowflakeCortex        Provider = "snowflake-cortex"
	ProviderSynthetic              Provider = "synthetics"
	ProviderTogetherAI             Provider = "togetherai"
	ProviderVenice                 Provider = "venice"
	ProviderVercel                 Provider = "vercel"
	ProviderZAI                    Provider = "zai"
	ProviderZenMux                 Provider = "zenmux"
	// ProviderGoogle is Google's Gemini generateContent backend. The provider (the
	// backend "google") and the dialect (APIFormatGemini) are distinct axes: google
	// speaks only the Gemini wire format, authenticated with an x-goog-api-key header
	// (RequiredAuth → AuthAPIKey), and is served by the bespoke providers/gemini
	// client (the generic transport assumes a static /chat/completions path).
	ProviderGoogle Provider = "google"
)

func (Provider) AuthPolicy

func (p Provider) AuthPolicy(format model.APIFormat) (AuthPolicy, error)

AuthPolicy is the exact-identity successor to RequiredAuth. RequiredAuth is retained as the source-compatible legacy bridge; new construction code should use this policy so provider, transport, scheme, usage, issuer, and audience are all bound together.

func (Provider) RequiredAuth

func (p Provider) RequiredAuth() (auth.AuthKind, error)

RequiredAuth reports which credential kind the provider needs, erroring on an unknown provider so a newly added one must be classified here before use. Multi-auth-ready successor to RequiresKey; fail-closed by the same rationale (a permissive default would fail open).

func (Provider) RequiresKey

func (p Provider) RequiresKey() (bool, error)

RequiresKey reports whether the provider needs an API key, and errors on an unknown provider so a newly added one must be classified here before it can be used. Hosted key providers (phala, chutes, openrouter, google) require a key; a local LM Studio endpoint does not. A bare default-false would fail open — the bug this method exists to prevent. This is the legacy boolean superseded by RequiredAuth (which is the real gate and can express non-key auth like SigV4).

Directories

Path Synopsis
Package aci implements a client for the Dstack private-ai-gateway "aci/1" confidential-inference protocol.
Package aci implements a client for the Dstack private-ai-gateway "aci/1" confidential-inference protocol.
Package auth provides the provider-specific AWS Signature Version 4 (SigV4) Authenticator for the inference client seam.
Package auth provides the provider-specific AWS Signature Version 4 (SigV4) Authenticator for the inference client seam.
Package auto is the composition root that selects and wires a concrete inference.Client for a validated Model.
Package auto is the composition root that selects and wires a concrete inference.Client for a validated Model.
Package e2e — shared end-to-end envelope primitives.
Package e2e — shared end-to-end envelope primitives.
examples
auto command
credentials command
internal
credentialclient
Package credentialclient adapts a credentials.Source to the call-scoped authorization seam exposed by inference transports.
Package credentialclient adapts a credentials.Source to the call-scoped authorization seam exposed by inference transports.
providers
anthropic
Package anthropic provides a native Anthropic Messages API client.
Package anthropic provides a native Anthropic Messages API client.
anthropic/subscription
Package subscription exposes the Anthropic subscription registration policy boundary.
Package subscription exposes the Anthropic subscription registration policy boundary.
atomic-chat
Package atomicchat provides the documented atomicchat OpenAI-compatible API.
Package atomicchat provides the documented atomicchat OpenAI-compatible API.
azure
Package azure provides an Azure OpenAI Responses API client.
Package azure provides an Azure OpenAI Responses API client.
azure-cognitive-services
Package azurecognitive provides Azure Cognitive Services' documented OpenAI and Anthropic-compatible model endpoints.
Package azurecognitive provides Azure Cognitive Services' documented OpenAI and Anthropic-compatible model endpoints.
baseten
Package baseten provides the documented baseten OpenAI-compatible API.
Package baseten provides the documented baseten OpenAI-compatible API.
bedrock
Package bedrock is an AWS Bedrock Runtime client for Bedrock InvokeModel and native Converse/ConverseStream.
Package bedrock is an AWS Bedrock Runtime client for Bedrock InvokeModel and native Converse/ConverseStream.
cerebras
Package cerebras provides the documented cerebras OpenAI-compatible API.
Package cerebras provides the documented cerebras OpenAI-compatible API.
chutes
Package chutes is a Chutes end-to-end-encrypted, TEE-attested LLM client.
Package chutes is a Chutes end-to-end-encrypted, TEE-attested LLM client.
cloudflare-ai-gateway
Package cloudflaregateway provides Cloudflare AI Gateway's documented OpenAI, Responses, and Anthropic proxy endpoints.
Package cloudflaregateway provides Cloudflare AI Gateway's documented OpenAI, Responses, and Anthropic proxy endpoints.
cloudflare-workers-ai
Package cloudflareworkers provides Cloudflare Workers AI's documented OpenAI-compatible AI Gateway route.
Package cloudflareworkers provides Cloudflare Workers AI's documented OpenAI-compatible AI Gateway route.
cortecs
Package cortecs provides the documented cortecs OpenAI-compatible API.
Package cortecs provides the documented cortecs OpenAI-compatible API.
deepinfra
Package deepinfra provides Deep Infra's documented OpenAI and Anthropic compatible endpoints.
Package deepinfra provides Deep Infra's documented OpenAI and Anthropic compatible endpoints.
deepseek
Package deepseek provides the documented deepseek OpenAI-compatible API.
Package deepseek provides the documented deepseek OpenAI-compatible API.
digitalocean
Package digitalocean provides the documented digitalocean OpenAI-compatible API.
Package digitalocean provides the documented digitalocean OpenAI-compatible API.
fireworks
Package fireworks provides the documented fireworks OpenAI-compatible API.
Package fireworks provides the documented fireworks OpenAI-compatible API.
frogbot
Package frogbot provides the documented frogbot OpenAI-compatible API.
Package frogbot provides the documented frogbot OpenAI-compatible API.
gemini
Package gemini is a bespoke client for Google's Gemini generateContent API.
Package gemini is a bespoke client for Google's Gemini generateContent API.
github-copilot
Package githubcopilot provides GitHub Copilot's documented OpenAI Chat and Responses gateway endpoints.
Package githubcopilot provides GitHub Copilot's documented OpenAI Chat and Responses gateway endpoints.
gitlab
Package gitlab provides GitLab Duo's documented AI Gateway proxy endpoints.
Package gitlab provides GitLab Duo's documented AI Gateway proxy endpoints.
gmicloud
Package gmicloud provides GMI Cloud's documented OpenAI-compatible Chat Completions API.
Package gmicloud provides GMI Cloud's documented OpenAI-compatible Chat Completions API.
google-vertex
Package vertex provides Google Vertex AI's documented Gemini generateContent route and Anthropic Claude rawPredict route.
Package vertex provides Google Vertex AI's documented Gemini generateContent route and Anthropic Claude rawPredict route.
groq
Package groq provides the documented groq OpenAI-compatible API.
Package groq provides the documented groq OpenAI-compatible API.
helicone
Package helicone provides the documented helicone OpenAI-compatible API.
Package helicone provides the documented helicone OpenAI-compatible API.
huggingface
Package huggingface provides the documented huggingface OpenAI-compatible API.
Package huggingface provides the documented huggingface OpenAI-compatible API.
internal/compat
Package compat contains the deliberately small shared core used by documented providers whose public endpoint is compatible with one of the bundled request codecs.
Package compat contains the deliberately small shared core used by documented providers whose public endpoint is compatible with one of the bundled request codecs.
internal/contracttest
Package contracttest contains deterministic provider-package HTTP contracts.
Package contracttest contains deterministic provider-package HTTP contracts.
internal/simple
Package simple contains the common adapter used by provider packages whose documented endpoint is OpenAI Chat-compatible and uses either bearer or no authentication.
Package simple contains the common adapter used by provider packages whose documented endpoint is OpenAI Chat-compatible and uses either bearer or no authentication.
internal/subscription
Package subscription contains provider-neutral, fixture-only contracts for credential-backed subscription transports.
Package subscription contains provider-neutral, fixture-only contracts for credential-backed subscription transports.
ionet
Package ionet provides the documented ionet OpenAI-compatible API.
Package ionet provides the documented ionet OpenAI-compatible API.
llama
Package llama provides Meta's hosted Llama API through its documented OpenAI-compatible Chat Completions endpoint.
Package llama provides Meta's hosted Llama API through its documented OpenAI-compatible Chat Completions endpoint.
llamacpp
Package llamacpp provides the local llama.cpp llama-server OpenAI-compatible API.
Package llamacpp provides the local llama.cpp llama-server OpenAI-compatible API.
llmgateway
Package llmgateway provides LLM Gateway's documented OpenAI Chat and Anthropic Messages proxy endpoints.
Package llmgateway provides LLM Gateway's documented OpenAI Chat and Anthropic Messages proxy endpoints.
minimax
Package minimax provides MiniMax's native Anthropic Messages endpoint.
Package minimax provides MiniMax's native Anthropic Messages endpoint.
moonshot
Package moonshot provides the documented moonshot OpenAI-compatible API.
Package moonshot provides the documented moonshot OpenAI-compatible API.
nebius
Package nebius provides the documented nebius OpenAI-compatible API.
Package nebius provides the documented nebius OpenAI-compatible API.
nvidia
Package nvidia provides the documented nvidia OpenAI-compatible API.
Package nvidia provides the documented nvidia OpenAI-compatible API.
ollama
Package ollama provides the documented ollama OpenAI-compatible API.
Package ollama provides the documented ollama OpenAI-compatible API.
ollamacloud
Package ollamacloud provides the documented ollamacloud OpenAI-compatible API.
Package ollamacloud provides the documented ollamacloud OpenAI-compatible API.
openai
Package openai provides OpenAI Chat Completions and Responses API clients.
Package openai provides OpenAI Chat Completions and Responses API clients.
openai/subscription
Package subscription exposes the OpenAI subscription registration policy boundary.
Package subscription exposes the OpenAI subscription registration policy boundary.
opencode
Package opencode provides the OpenCode Zen OpenAI-compatible endpoint.
Package opencode provides the OpenCode Zen OpenAI-compatible endpoint.
opencode-go
Package opencode provides the OpenCode Go endpoint using the shared Chat, Responses, and Anthropic transport semantics.
Package opencode provides the OpenCode Go endpoint using the shared Chat, Responses, and Anthropic transport semantics.
openrouter
Package openrouter provides the OpenRouter-specific construction and request options for the OpenAI-compatible Chat Completions API.
Package openrouter provides the OpenRouter-specific construction and request options for the OpenAI-compatible Chat Completions API.
ovhcloud
Package ovhcloud provides the documented ovhcloud OpenAI-compatible API.
Package ovhcloud provides the documented ovhcloud OpenAI-compatible API.
p302ai
Package p302ai provides the 302.AI OpenAI-compatible Chat Completions API.
Package p302ai provides the 302.AI OpenAI-compatible Chat Completions API.
phala
Package phala is the Phala confidential-inference provider: it owns the gateway base-URL default and a typed constructor (New) that wires the reusable, provider-agnostic aci attestation protocol into an attested inference.Client.
Package phala is the Phala confidential-inference provider: it owns the gateway base-URL default and a typed constructor (New) that wires the reusable, provider-agnostic aci attestation protocol into an attested inference.Client.
sap-ai-core
Package sap provides SAP AI Core's orchestration chat endpoint.
Package sap provides SAP AI Core's orchestration chat endpoint.
scaleway
Package scaleway provides the documented scaleway OpenAI-compatible API.
Package scaleway provides the documented scaleway OpenAI-compatible API.
snowflake-cortex
Package snowflake provides Snowflake Cortex's documented OpenAI-compatible Chat Completions endpoint.
Package snowflake provides Snowflake Cortex's documented OpenAI-compatible Chat Completions endpoint.
stackit
Package stackit provides the documented stackit OpenAI-compatible API.
Package stackit provides the documented stackit OpenAI-compatible API.
synthetic
Package synthetic provides the documented Synthetic.new OpenAI-compatible API.
Package synthetic provides the documented Synthetic.new OpenAI-compatible API.
together
Package together provides the documented together OpenAI-compatible API.
Package together provides the documented together OpenAI-compatible API.
venice
Package venice provides Venice AI's documented OpenAI Chat and Responses endpoints.
Package venice provides Venice AI's documented OpenAI Chat and Responses endpoints.
vercel
Package vercel provides Vercel AI Gateway's documented OpenAI Chat, Responses, and Anthropic Messages endpoints.
Package vercel provides Vercel AI Gateway's documented OpenAI Chat, Responses, and Anthropic Messages endpoints.
xai
Package xai provides xAI Chat Completions and Responses API clients.
Package xai provides xAI Chat Completions and Responses API clients.
zai
Package zai provides the documented zai OpenAI-compatible API.
Package zai provides the documented zai OpenAI-compatible API.
zenmux
Package zenmux provides ZenMux's documented OpenAI, Responses, and Anthropic protocol-conversion endpoints.
Package zenmux provides ZenMux's documented OpenAI, Responses, and Anthropic protocol-conversion endpoints.
Package tee — shared TEE attestation primitives.
Package tee — shared TEE attestation primitives.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL