Documentation
¶
Overview ¶
Package llm is the batteries-included provider SDK layered on the neutral github.com/looprig/inference model-call contract. It owns the provider POLICY that inference deliberately does not carry: the known-provider registry, the provider/API-format truth table, provider auth requirements, provider default endpoint behavior, and the fail-closed model-validation preset. It also hosts the self-contained provider-security machinery (aci, e2e, tee) and the provider-specific SigV4 authenticator (llm/auth). inference never imports llm.
Index ¶
- Constants
- func ValidateModel(m model.Model) error
- type AttestationError
- type AuthBinding
- type AuthPolicy
- type AuthPolicyMismatchError
- type AuthRequiredError
- type CounterConstructor
- type CounterDirectConstructionError
- type CounterDirectConstructionReason
- type CounterSupportError
- type CounterSupportReason
- type InvalidAuthPolicyError
- type Provider
Constants ¶
const APIFormatBedrockConverse = model.APIFormatBedrockConverse
APIFormatBedrockConverse is retained as a provider-module alias for the neutral Bedrock Converse wire-dialect label owned by inference/model.
const AuthGCP auth.AuthKind = "gcp"
AuthGCP classifies Google Cloud application credentials or an exchanged Vertex access token. Vertex construction is direct because auto.New cannot safely discover or refresh ADC credentials from its APIKey input.
const AuthOAuth auth.AuthKind = "oauth"
AuthOAuth classifies a provider credential obtained through an OAuth flow. The auto factory receives only an APIKey-shaped secret and therefore treats OAuth providers as direct-construction providers until a caller supplies the exchanged access token explicitly.
const AuthServiceKey auth.AuthKind = "service_key"
AuthServiceKey classifies a provider service-key document that contains more than one credential or endpoint field.
const AuthSigV4 auth.AuthKind = "sigv4"
AuthSigV4 classifies the AWS Signature Version 4 credential a Bedrock provider requires. It is an auth.AuthKind value defined here rather than in inference: inference owns only the generic credential kinds (AuthNone, AuthAPIKey), while provider-specific request-signing schemes are llm policy.
const AuthToken auth.AuthKind = "token"
AuthToken classifies a bearer/PAT credential whose provider also needs a separate account or tenant identifier.
Variables ¶
This section is empty.
Functions ¶
func ValidateModel ¶
ValidateModel is the fail-closed provider-policy preset that layers the known-provider truth table on top of model.Model's structural validation. model.Model.Validate is deliberately provider-policy-free (it only checks a non-empty Name and a syntactically safe non-empty BaseURL); this preset adds the checks inference dropped:
- Provider must be a known backend (RequiredAuth errors on an unclassified one).
- Provider must speak the model's APIFormat (fail-closed: an unsupported pair is rejected).
- An empty BaseURL is only acceptable when the provider resolves a canonical or region-routed endpoint (allowsEmptyBaseURL); otherwise an explicit base is required.
It reproduces the pre-split harness Model.Validate behavior, returning a *model.ValidationError on the first rule violated. OriginCustom models validate identically to catalog rows; the lower trust in a custom model's Caps is a downstream gating concern, not this preset's.
Types ¶
type AttestationError ¶
AttestationError is a TEE attestation failure. Fail-closed: a request must never be sent to the provider when this error is returned. Err may be nil when the failure has no underlying cause to chain.
It lives in llm, not inference: attestation is provider-security policy (Phala/Chutes confidential inference), which the neutral model-call layer deliberately does not carry.
func (*AttestationError) Error ¶
func (e *AttestationError) Error() string
func (*AttestationError) Unwrap ¶
func (e *AttestationError) Unwrap() error
type AuthBinding ¶
type AuthBinding struct {
Provider string
Transport string
Scheme credentials.Scheme
Usage credentials.UsageClass
Issuer string
Audience string
}
AuthBinding is the complete authority identity accepted by one provider transport. Scheme and usage are intentionally not enough to identify a credential: provider, transport, issuer, and audience are part of the binding as well.
func (AuthBinding) Descriptor ¶
func (b AuthBinding) Descriptor() (credentials.Descriptor, error)
Descriptor converts the policy binding into the credential descriptor used by a source and its leases. Label is presentation metadata and is therefore deliberately absent from an AuthBinding.
func (AuthBinding) Valid ¶
func (b AuthBinding) Valid() bool
Valid reports whether the complete binding can be represented by a safe credential descriptor.
type AuthPolicy ¶
type AuthPolicy struct {
Accepted []AuthBinding
}
AuthPolicy is the immutable set of complete credential identities accepted by one provider client. A policy with no accepted bindings is invalid.
func AuthPolicyForModel ¶
func AuthPolicyForModel(selected model.Model) (AuthPolicy, error)
AuthPolicyForModel is the one-way compatibility bridge from the legacy Provider.RequiredAuth registry to an exact credentials policy. It is kept in llm because inference must remain unaware of provider identity and policy.
func (AuthPolicy) Accepts ¶
func (p AuthPolicy) Accepts(descriptor credentials.Descriptor) bool
Accepts reports whether descriptor has exactly one of the policy's accepted six-field identities. Descriptor.Label is presentation-only and is ignored.
func (AuthPolicy) Match ¶
func (p AuthPolicy) Match(descriptor credentials.Descriptor) error
Match validates descriptor against the policy and returns a safe typed mismatch error when it is not accepted.
func (AuthPolicy) Validate ¶
func (p AuthPolicy) Validate() error
Validate checks every accepted identity and rejects duplicate bindings. The input slice is never mutated; callers should treat a policy as immutable once it has been used to construct a client.
type AuthPolicyMismatchError ¶
type AuthPolicyMismatchError struct{ Reason string }
AuthPolicyMismatchError reports a source or lease identity that is not authorized for the bound provider transport.
func (*AuthPolicyMismatchError) Error ¶
func (e *AuthPolicyMismatchError) Error() string
type AuthRequiredError ¶
AuthRequiredError is returned by a provider factory when a provider that requires credentials is given none. Fail-closed. Carries no secret. Provider is the llm provider-policy label; Kind is the inference credential kind (including AuthSigV4).
func (*AuthRequiredError) Error ¶
func (e *AuthRequiredError) Error() string
type CounterConstructor ¶
type CounterConstructor string
CounterConstructor names a provider constructor without making it executable. It is a directive safe to expose in logs and user-facing configuration errors.
const (
CounterConstructorBedrock CounterConstructor = "bedrock.NewCounter"
)
type CounterDirectConstructionError ¶
type CounterDirectConstructionError struct {
Provider Provider
Reason CounterDirectConstructionReason
Use CounterConstructor
}
CounterDirectConstructionError directs callers to an exact provider counter constructor whose required inputs cannot be supplied by auto.NewCounter.
func (*CounterDirectConstructionError) Error ¶
func (e *CounterDirectConstructionError) Error() string
type CounterDirectConstructionReason ¶
type CounterDirectConstructionReason string
CounterDirectConstructionReason classifies why auto.NewCounter cannot build an exact counter from its (Model, APIKey) inputs.
const ( // CounterDirectConstructionNeedsSigV4 means construction requires AWS SigV4 // credentials and a region, neither of which can be represented by APIKey. CounterDirectConstructionNeedsSigV4 CounterDirectConstructionReason = "requires AWS SigV4 credentials and region" )
type CounterSupportError ¶
type CounterSupportError struct {
Provider Provider
Reason CounterSupportReason
APIFormat model.APIFormat
}
CounterSupportError reports that a known provider has no exact context counter in llm for the selected dialect. All fields are secret-free and inspectable with errors.As.
func (*CounterSupportError) Error ¶
func (e *CounterSupportError) Error() string
type CounterSupportReason ¶
type CounterSupportReason string
CounterSupportReason classifies why an exact provider context counter is not available through the llm module. It never contains provider-controlled text.
const ( // that llm can construct. Consumers may explicitly inject a local estimator; // this error never substitutes one. CounterSupportExactUnavailable CounterSupportReason = "exact provider context counter unavailable" // but it cannot encode the model's validated API dialect. CounterSupportAPIFormatUnavailable CounterSupportReason = "exact provider context counter unavailable for API format" )
type InvalidAuthPolicyError ¶
InvalidAuthPolicyError reports malformed provider policy metadata. It carries no credential or provider response data.
func (*InvalidAuthPolicyError) Error ¶
func (e *InvalidAuthPolicyError) Error() string
func (*InvalidAuthPolicyError) Unwrap ¶
func (e *InvalidAuthPolicyError) Unwrap() error
type Provider ¶
type Provider string
Provider names the concrete backend an llm/auto factory dispatches on. Unknown values are rejected by ValidateModel; a provider constructor additionally enforces each provider's auth requirement. It is the provider-policy analogue of the opaque model.ProviderName label: inference carries no provider constants, no auth requirements, and no default endpoints — those live here.
const ( ProviderLMStudio Provider = "lmstudio" ProviderPhala Provider = "phala" ProviderChutes Provider = "chutes" ProviderOpenRouter Provider = "openrouter" ProviderOpenAI Provider = "openai" ProviderAzure Provider = "azure" ProviderAzureCognitiveServices Provider = "azure-cognitive-services" ProviderAnthropic Provider = "anthropic" ProviderXAI Provider = "xai" ProviderBedrock Provider = "bedrock" Provider302AI Provider = "302ai" ProviderAtomicChat Provider = "atomic-chat" ProviderBaseten Provider = "baseten" ProviderCerebras Provider = "cerebras" ProviderCloudflareAIGateway Provider = "cloudflare-ai-gateway" ProviderCloudflareWorkersAI Provider = "cloudflare-workers-ai" ProviderCortecs Provider = "cortecs" ProviderDeepSeek Provider = "deepseek" ProviderDeepInfra Provider = "deepinfra" ProviderDigitalOcean Provider = "digitalocean" ProviderFrogBot Provider = "frogbot" ProviderFireworks Provider = "fireworks-ai" ProviderGitLab Provider = "gitlab" ProviderGitHubCopilot Provider = "github-copilot" ProviderGMICloud Provider = "gmicloud" ProviderGoogleVertex Provider = "google-vertex" ProviderGoogleVertexAnthropic Provider = "google-vertex-anthropic" ProviderGroq Provider = "groq" ProviderHuggingFace Provider = "huggingface" ProviderHelicone Provider = "helicone" ProviderLlama Provider = "llama" ProviderLlamaCPP Provider = "llama.cpp" ProviderIONet Provider = "io-net" ProviderMoonshot Provider = "moonshotai" ProviderMiniMax Provider = "minimax" ProviderNVIDIA Provider = "nvidia" ProviderNebius Provider = "nebius" ProviderOllama Provider = "ollama" ProviderOllamaCloud Provider = "ollama-cloud" ProviderOpenCode Provider = "opencode" ProviderOpenCodeGo Provider = "opencode-go" ProviderLLMGateway Provider = "llmgateway" ProviderSAP Provider = "sap-ai-core" ProviderSTACKIT Provider = "stackit" ProviderOVHCloud Provider = "ovhcloud" ProviderScaleway Provider = "scaleway" ProviderSnowflakeCortex Provider = "snowflake-cortex" ProviderSynthetic Provider = "synthetics" ProviderTogetherAI Provider = "togetherai" ProviderVenice Provider = "venice" ProviderVercel Provider = "vercel" ProviderZAI Provider = "zai" ProviderZenMux Provider = "zenmux" // ProviderGoogle is Google's Gemini generateContent backend. The provider (the // backend "google") and the dialect (APIFormatGemini) are distinct axes: google // speaks only the Gemini wire format, authenticated with an x-goog-api-key header // (RequiredAuth → AuthAPIKey), and is served by the bespoke providers/gemini // client (the generic transport assumes a static /chat/completions path). ProviderGoogle Provider = "google" )
func (Provider) AuthPolicy ¶
func (p Provider) AuthPolicy(format model.APIFormat) (AuthPolicy, error)
AuthPolicy is the exact-identity successor to RequiredAuth. RequiredAuth is retained as the source-compatible legacy bridge; new construction code should use this policy so provider, transport, scheme, usage, issuer, and audience are all bound together.
func (Provider) RequiredAuth ¶
RequiredAuth reports which credential kind the provider needs, erroring on an unknown provider so a newly added one must be classified here before use. Multi-auth-ready successor to RequiresKey; fail-closed by the same rationale (a permissive default would fail open).
func (Provider) RequiresKey ¶
RequiresKey reports whether the provider needs an API key, and errors on an unknown provider so a newly added one must be classified here before it can be used. Hosted key providers (phala, chutes, openrouter, google) require a key; a local LM Studio endpoint does not. A bare default-false would fail open — the bug this method exists to prevent. This is the legacy boolean superseded by RequiredAuth (which is the real gate and can express non-key auth like SigV4).
Directories
¶
| Path | Synopsis |
|---|---|
|
Package aci implements a client for the Dstack private-ai-gateway "aci/1" confidential-inference protocol.
|
Package aci implements a client for the Dstack private-ai-gateway "aci/1" confidential-inference protocol. |
|
Package auth provides the provider-specific AWS Signature Version 4 (SigV4) Authenticator for the inference client seam.
|
Package auth provides the provider-specific AWS Signature Version 4 (SigV4) Authenticator for the inference client seam. |
|
Package auto is the composition root that selects and wires a concrete inference.Client for a validated Model.
|
Package auto is the composition root that selects and wires a concrete inference.Client for a validated Model. |
|
Package e2e — shared end-to-end envelope primitives.
|
Package e2e — shared end-to-end envelope primitives. |
|
examples
|
|
|
auto
command
|
|
|
credentials
command
|
|
|
internal
|
|
|
credentialclient
Package credentialclient adapts a credentials.Source to the call-scoped authorization seam exposed by inference transports.
|
Package credentialclient adapts a credentials.Source to the call-scoped authorization seam exposed by inference transports. |
|
providers
|
|
|
anthropic
Package anthropic provides a native Anthropic Messages API client.
|
Package anthropic provides a native Anthropic Messages API client. |
|
anthropic/subscription
Package subscription exposes the Anthropic subscription registration policy boundary.
|
Package subscription exposes the Anthropic subscription registration policy boundary. |
|
atomic-chat
Package atomicchat provides the documented atomicchat OpenAI-compatible API.
|
Package atomicchat provides the documented atomicchat OpenAI-compatible API. |
|
azure
Package azure provides an Azure OpenAI Responses API client.
|
Package azure provides an Azure OpenAI Responses API client. |
|
azure-cognitive-services
Package azurecognitive provides Azure Cognitive Services' documented OpenAI and Anthropic-compatible model endpoints.
|
Package azurecognitive provides Azure Cognitive Services' documented OpenAI and Anthropic-compatible model endpoints. |
|
baseten
Package baseten provides the documented baseten OpenAI-compatible API.
|
Package baseten provides the documented baseten OpenAI-compatible API. |
|
bedrock
Package bedrock is an AWS Bedrock Runtime client for Bedrock InvokeModel and native Converse/ConverseStream.
|
Package bedrock is an AWS Bedrock Runtime client for Bedrock InvokeModel and native Converse/ConverseStream. |
|
cerebras
Package cerebras provides the documented cerebras OpenAI-compatible API.
|
Package cerebras provides the documented cerebras OpenAI-compatible API. |
|
chutes
Package chutes is a Chutes end-to-end-encrypted, TEE-attested LLM client.
|
Package chutes is a Chutes end-to-end-encrypted, TEE-attested LLM client. |
|
cloudflare-ai-gateway
Package cloudflaregateway provides Cloudflare AI Gateway's documented OpenAI, Responses, and Anthropic proxy endpoints.
|
Package cloudflaregateway provides Cloudflare AI Gateway's documented OpenAI, Responses, and Anthropic proxy endpoints. |
|
cloudflare-workers-ai
Package cloudflareworkers provides Cloudflare Workers AI's documented OpenAI-compatible AI Gateway route.
|
Package cloudflareworkers provides Cloudflare Workers AI's documented OpenAI-compatible AI Gateway route. |
|
cortecs
Package cortecs provides the documented cortecs OpenAI-compatible API.
|
Package cortecs provides the documented cortecs OpenAI-compatible API. |
|
deepinfra
Package deepinfra provides Deep Infra's documented OpenAI and Anthropic compatible endpoints.
|
Package deepinfra provides Deep Infra's documented OpenAI and Anthropic compatible endpoints. |
|
deepseek
Package deepseek provides the documented deepseek OpenAI-compatible API.
|
Package deepseek provides the documented deepseek OpenAI-compatible API. |
|
digitalocean
Package digitalocean provides the documented digitalocean OpenAI-compatible API.
|
Package digitalocean provides the documented digitalocean OpenAI-compatible API. |
|
fireworks
Package fireworks provides the documented fireworks OpenAI-compatible API.
|
Package fireworks provides the documented fireworks OpenAI-compatible API. |
|
frogbot
Package frogbot provides the documented frogbot OpenAI-compatible API.
|
Package frogbot provides the documented frogbot OpenAI-compatible API. |
|
gemini
Package gemini is a bespoke client for Google's Gemini generateContent API.
|
Package gemini is a bespoke client for Google's Gemini generateContent API. |
|
github-copilot
Package githubcopilot provides GitHub Copilot's documented OpenAI Chat and Responses gateway endpoints.
|
Package githubcopilot provides GitHub Copilot's documented OpenAI Chat and Responses gateway endpoints. |
|
gitlab
Package gitlab provides GitLab Duo's documented AI Gateway proxy endpoints.
|
Package gitlab provides GitLab Duo's documented AI Gateway proxy endpoints. |
|
gmicloud
Package gmicloud provides GMI Cloud's documented OpenAI-compatible Chat Completions API.
|
Package gmicloud provides GMI Cloud's documented OpenAI-compatible Chat Completions API. |
|
google-vertex
Package vertex provides Google Vertex AI's documented Gemini generateContent route and Anthropic Claude rawPredict route.
|
Package vertex provides Google Vertex AI's documented Gemini generateContent route and Anthropic Claude rawPredict route. |
|
groq
Package groq provides the documented groq OpenAI-compatible API.
|
Package groq provides the documented groq OpenAI-compatible API. |
|
helicone
Package helicone provides the documented helicone OpenAI-compatible API.
|
Package helicone provides the documented helicone OpenAI-compatible API. |
|
huggingface
Package huggingface provides the documented huggingface OpenAI-compatible API.
|
Package huggingface provides the documented huggingface OpenAI-compatible API. |
|
internal/compat
Package compat contains the deliberately small shared core used by documented providers whose public endpoint is compatible with one of the bundled request codecs.
|
Package compat contains the deliberately small shared core used by documented providers whose public endpoint is compatible with one of the bundled request codecs. |
|
internal/contracttest
Package contracttest contains deterministic provider-package HTTP contracts.
|
Package contracttest contains deterministic provider-package HTTP contracts. |
|
internal/simple
Package simple contains the common adapter used by provider packages whose documented endpoint is OpenAI Chat-compatible and uses either bearer or no authentication.
|
Package simple contains the common adapter used by provider packages whose documented endpoint is OpenAI Chat-compatible and uses either bearer or no authentication. |
|
internal/subscription
Package subscription contains provider-neutral, fixture-only contracts for credential-backed subscription transports.
|
Package subscription contains provider-neutral, fixture-only contracts for credential-backed subscription transports. |
|
ionet
Package ionet provides the documented ionet OpenAI-compatible API.
|
Package ionet provides the documented ionet OpenAI-compatible API. |
|
llama
Package llama provides Meta's hosted Llama API through its documented OpenAI-compatible Chat Completions endpoint.
|
Package llama provides Meta's hosted Llama API through its documented OpenAI-compatible Chat Completions endpoint. |
|
llamacpp
Package llamacpp provides the local llama.cpp llama-server OpenAI-compatible API.
|
Package llamacpp provides the local llama.cpp llama-server OpenAI-compatible API. |
|
llmgateway
Package llmgateway provides LLM Gateway's documented OpenAI Chat and Anthropic Messages proxy endpoints.
|
Package llmgateway provides LLM Gateway's documented OpenAI Chat and Anthropic Messages proxy endpoints. |
|
minimax
Package minimax provides MiniMax's native Anthropic Messages endpoint.
|
Package minimax provides MiniMax's native Anthropic Messages endpoint. |
|
moonshot
Package moonshot provides the documented moonshot OpenAI-compatible API.
|
Package moonshot provides the documented moonshot OpenAI-compatible API. |
|
nebius
Package nebius provides the documented nebius OpenAI-compatible API.
|
Package nebius provides the documented nebius OpenAI-compatible API. |
|
nvidia
Package nvidia provides the documented nvidia OpenAI-compatible API.
|
Package nvidia provides the documented nvidia OpenAI-compatible API. |
|
ollama
Package ollama provides the documented ollama OpenAI-compatible API.
|
Package ollama provides the documented ollama OpenAI-compatible API. |
|
ollamacloud
Package ollamacloud provides the documented ollamacloud OpenAI-compatible API.
|
Package ollamacloud provides the documented ollamacloud OpenAI-compatible API. |
|
openai
Package openai provides OpenAI Chat Completions and Responses API clients.
|
Package openai provides OpenAI Chat Completions and Responses API clients. |
|
openai/subscription
Package subscription exposes the OpenAI subscription registration policy boundary.
|
Package subscription exposes the OpenAI subscription registration policy boundary. |
|
opencode
Package opencode provides the OpenCode Zen OpenAI-compatible endpoint.
|
Package opencode provides the OpenCode Zen OpenAI-compatible endpoint. |
|
opencode-go
Package opencode provides the OpenCode Go endpoint using the shared Chat, Responses, and Anthropic transport semantics.
|
Package opencode provides the OpenCode Go endpoint using the shared Chat, Responses, and Anthropic transport semantics. |
|
openrouter
Package openrouter provides the OpenRouter-specific construction and request options for the OpenAI-compatible Chat Completions API.
|
Package openrouter provides the OpenRouter-specific construction and request options for the OpenAI-compatible Chat Completions API. |
|
ovhcloud
Package ovhcloud provides the documented ovhcloud OpenAI-compatible API.
|
Package ovhcloud provides the documented ovhcloud OpenAI-compatible API. |
|
p302ai
Package p302ai provides the 302.AI OpenAI-compatible Chat Completions API.
|
Package p302ai provides the 302.AI OpenAI-compatible Chat Completions API. |
|
phala
Package phala is the Phala confidential-inference provider: it owns the gateway base-URL default and a typed constructor (New) that wires the reusable, provider-agnostic aci attestation protocol into an attested inference.Client.
|
Package phala is the Phala confidential-inference provider: it owns the gateway base-URL default and a typed constructor (New) that wires the reusable, provider-agnostic aci attestation protocol into an attested inference.Client. |
|
sap-ai-core
Package sap provides SAP AI Core's orchestration chat endpoint.
|
Package sap provides SAP AI Core's orchestration chat endpoint. |
|
scaleway
Package scaleway provides the documented scaleway OpenAI-compatible API.
|
Package scaleway provides the documented scaleway OpenAI-compatible API. |
|
snowflake-cortex
Package snowflake provides Snowflake Cortex's documented OpenAI-compatible Chat Completions endpoint.
|
Package snowflake provides Snowflake Cortex's documented OpenAI-compatible Chat Completions endpoint. |
|
stackit
Package stackit provides the documented stackit OpenAI-compatible API.
|
Package stackit provides the documented stackit OpenAI-compatible API. |
|
synthetic
Package synthetic provides the documented Synthetic.new OpenAI-compatible API.
|
Package synthetic provides the documented Synthetic.new OpenAI-compatible API. |
|
together
Package together provides the documented together OpenAI-compatible API.
|
Package together provides the documented together OpenAI-compatible API. |
|
venice
Package venice provides Venice AI's documented OpenAI Chat and Responses endpoints.
|
Package venice provides Venice AI's documented OpenAI Chat and Responses endpoints. |
|
vercel
Package vercel provides Vercel AI Gateway's documented OpenAI Chat, Responses, and Anthropic Messages endpoints.
|
Package vercel provides Vercel AI Gateway's documented OpenAI Chat, Responses, and Anthropic Messages endpoints. |
|
xai
Package xai provides xAI Chat Completions and Responses API clients.
|
Package xai provides xAI Chat Completions and Responses API clients. |
|
zai
Package zai provides the documented zai OpenAI-compatible API.
|
Package zai provides the documented zai OpenAI-compatible API. |
|
zenmux
Package zenmux provides ZenMux's documented OpenAI, Responses, and Anthropic protocol-conversion endpoints.
|
Package zenmux provides ZenMux's documented OpenAI, Responses, and Anthropic protocol-conversion endpoints. |
|
Package tee — shared TEE attestation primitives.
|
Package tee — shared TEE attestation primitives. |