Documentation
¶
Index ¶
- Constants
- Variables
- func CanUseGPU(context.Context, string) (bool, error)
- func ExtractLlamaCppError(output string) string
- func GetContextSize(modelCfg types.ModelConfig, backendCfg *inference.BackendConfiguration) *int32
- func GetDesiredServerVersion() string
- func GetReasoningBudget(backendCfg *inference.BackendConfiguration) *int32
- func NeedsDeferredInstall() bool
- func New(log logging.Logger, modelManager *models.Manager, serverLog logging.Logger, ...) (inference.Backend, error)
- func SetDesiredServerVersion(version string)
- type Config
Constants ¶
const ( // LatestServerVersion is the sentinel that opts into tracking the mutable // "latest" tag rather than a pinned release. LatestServerVersion = "latest" )
const (
// Name is the backend name.
Name = "llama.cpp"
)
const UnlimitedContextSize = -1
Variables ¶
Functions ¶
func ExtractLlamaCppError ¶ added in v1.1.10
ExtractLlamaCppError attempts to extract a meaningful error message from llama.cpp output. It looks for common error patterns and returns a cleaner, more user-friendly message alongside the original verbose output for easier debugging. The verbose output is sanitized to prevent leaking sensitive paths and truncated if it exceeds a reasonable length. If no recognizable pattern is found, it returns the full output.
func GetContextSize ¶
func GetContextSize(modelCfg types.ModelConfig, backendCfg *inference.BackendConfiguration) *int32
func GetDesiredServerVersion ¶
func GetDesiredServerVersion() string
func GetReasoningBudget ¶ added in v1.0.7
func GetReasoningBudget(backendCfg *inference.BackendConfiguration) *int32
func NeedsDeferredInstall ¶ added in v1.2.6
func NeedsDeferredInstall() bool
NeedsDeferredInstall reports whether the llama.cpp backend downloads its binary on demand on the current platform (macOS/Windows) rather than relying on a binary bundled at build time (Linux container image).
func New ¶
func New( log logging.Logger, modelManager *models.Manager, serverLog logging.Logger, installDir string, conf config.BackendConfig, registryMirrors []string, commandModifier func(*exec.Cmd), ) (inference.Backend, error)
New creates a new llama.cpp-based backend. installDir is the directory that holds (or, on macOS/Windows, will hold) the llama.cpp server binary. When it is empty a default writable location under the user's home directory is used.
func SetDesiredServerVersion ¶
func SetDesiredServerVersion(version string)
Types ¶
type Config ¶
type Config struct {
// Args are the base arguments that are always included.
Args []string
}
Config is the configuration for the llama.cpp backend.
func NewDefaultLlamaCppConfig ¶
func NewDefaultLlamaCppConfig() *Config
NewDefaultLlamaCppConfig creates a new LlamaCppConfig with default values.
func (*Config) GetArgs ¶
func (c *Config) GetArgs(bundle types.ModelBundle, socket string, mode inference.BackendMode, config *inference.BackendConfiguration) ([]string, error)
GetArgs implements BackendConfig.GetArgs.
Source Files