Why Go
- Case Studies
  
  Common problems companies solve with Go
- Use Cases
  
  Stories about how and why companies use Go
- Security
  
  How Go can help keep you secure by default
Learn
Docs
- Effective Go
  
  Tips for writing clear, performant, and idiomatic Go code
- Go User Manual
  
  A complete introduction to building software with Go
- Standard library
  
  Reference documentation for Go's standard library
- Release Notes
  
  Learn what's new in each Go release
Packages
Community
- Recorded Talks
  
  Videos from prior events
- Meetups
  
  Meet other local Go developers
- Conferences
  
  Learn and network with Go developers from around the world
- Go blog
  
  The Go project's official blog.
- Go project
  
  Get help and stay informed from Go
- Get connected

inference

package

Go to main page

Versions in this module

v1

v1.0.10

Dec 17, 2025

Changes in this version

+ func ValidateRuntimeFlags(flags []string) error

type BackendMode

+ func (m *BackendMode) UnmarshalJSON(data []byte) error

+ func (m BackendMode) MarshalJSON() ([]byte, error)

v1.0.9

Dec 11, 2025

v1.0.8

Dec 11, 2025

Changes in this version

type BackendMode

+ func ParseBackendMode(mode string) (BackendMode, bool)

type VLLMConfig

+ GPUMemoryUtilization *float64

v1.0.7

Dec 5, 2025

Changes in this version

type BackendConfiguration

+ LlamaCpp *LlamaCppConfig

+ VLLM *VLLMConfig

+ type HFOverrides map[string]interface

+ func (h HFOverrides) Validate() error

+ type LlamaCppConfig struct

+ ReasoningBudget *int64

+ type VLLMConfig struct

+ HFOverrides HFOverrides

v1.0.6

Nov 27, 2025

Changes in this version

+ const OriginOllamaCompletion

+ const RequestOriginHeader

v1.0.5

Nov 24, 2025

v1.0.4

Nov 20, 2025

Changes in this version

type BackendMode

+ const BackendModeReranking

v1.0.3

Nov 12, 2025

v1.0.2

Nov 6, 2025

v1.0.0

Nov 5, 2025

Changes in this version

+ const ExperimentalEndpointsPrefix

+ var InferencePrefix = "/engines"

+ var ModelsPrefix = "/models"

+ type Backend interface

+ GetDiskUsage func() (int64, error)

+ GetRequiredMemoryForModel func(ctx context.Context, model string, config *BackendConfiguration) (RequiredMemory, error)

+ Install func(ctx context.Context, httpClient *http.Client) error

+ Name func() string

+ Run func(ctx context.Context, socket, model string, modelRef string, mode BackendMode, ...) error

+ Status func() string

+ UsesExternalModelManagement func() bool

+ type BackendConfiguration struct

+ ContextSize int64

+ RuntimeFlags []string

+ Speculative *SpeculativeDecodingConfig

+ type BackendMode uint8

+ const BackendModeCompletion

+ const BackendModeEmbedding

+ func (m BackendMode) String() string

+ type ErrGGUFParse struct

+ Err error

+ func (e *ErrGGUFParse) Error() string

+ type RequiredMemory struct

+ RAM uint64

+ VRAM uint64

+ type SpeculativeDecodingConfig struct

+ DraftModel string

+ MinAcceptanceRate float64

+ NumTokens int