llm

package
v0.40.13 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 28, 2026 License: MIT Imports: 41 Imported by: 8

Documentation

Overview

llama_server.go wraps the llama-server binary as a subprocess

Ollama uses two chat paths with llama-server. Models with explicit Ollama renderers/parsers, Harmony handling, MLX, or an enabled Go TEMPLATE layer still render prompts in Go and call /completion. Other GGUF chat models use llama-server's chat_template handling through /v1/chat/completions.

For structured output, a JSON schema is passed to llama-server via its json_schema field and the "json" format as a builtin grammar via the grammar field. A format that applies after a response's thinking is sent as a grammar built around the GBNF llama-server itself derives from the schema.

llama-server auto-detects GPU layers (-ngl), thread count (-t), and flash attention (--flash-attn).

Index

Constants

View Source
const (
	DefaultEmbeddingNumBatch = 2048
)

DefaultEmbeddingNumBatch is the default NumBatch used for embedding models when neither the model nor the request specifies num_batch.

View Source
const SpeechSampleRate = 24000

SpeechSampleRate is the rate of the PCM a Speaker streams.

Variables

View Source
var ErrLoadRequiredFull = errors.New("unable to load full model on GPU")
View Source
var LlamaServerSysProcAttr = &syscall.SysProcAttr{}

Functions

func AddRunnerToJob added in v0.40.6

func AddRunnerToJob(int) error

AddRunnerToJob is a no-op on platforms without Windows Job Objects.

func AudioFormat added in v0.30.0

func AudioFormat(data []byte) (string, bool)

func FindLlamaServer added in v0.30.0

func FindLlamaServer() (string, error)

FindLlamaServer locates the llama-server binary in lib/ollama/. There is a single binary that dynamically loads GPU backends at runtime.

func IsLlamaCPP added in v0.40.0

func IsLlamaCPP(s LlamaServer) bool

IsLlamaCPP reports whether the server is backed by the llama.cpp llama-server runner implementation.

func IsOutOfMemory added in v0.30.0

func IsOutOfMemory(err error) bool

func IsOutOfMemoryMessage added in v0.30.0

func IsOutOfMemoryMessage(msg string) bool

func LlamaServerBatch added in v0.40.4

func LlamaServerBatch(f *gguf.Model, opts api.Options, completion bool) (int, uint64)

LlamaServerBatch resolves the batch size and extra memory allowance used when deciding whether to load a model alongside other models.

func LlamaServerBatchMemory added in v0.40.4

func LlamaServerBatchMemory(batch int) uint64

LlamaServerBatchMemory is the conservative extra memory allowance for a generation or whole-prompt decision batch above the default batch size.

func LlamaServerFlashAttention added in v0.30.0

func LlamaServerFlashAttention(gpus []ml.DeviceInfo) ml.FlashAttentionType

LlamaServerFlashAttention resolves the flash-attention mode passed to llama-server.

func LoadModel added in v0.1.33

func LoadModel(model string, maxArraySize int, shards ...string) (*gguf.Model, error)

LoadModel loads GGUF model metadata from disk.

It collects array values for arrays with a size less than or equal to maxArraySize. If maxArraySize is 0, the default value of 1024 is used. If the maxArraySize is negative, all arrays are collected.

func PredictServerVRAM added in v0.30.0

func PredictServerVRAM(modelPath string, f *gguf.Model, numCtx int) uint64

PredictServerVRAM estimates VRAM usage for a model without spawning llama-server. Uses model file size as a proxy for weights plus a rough KV cache estimate. This is intentionally conservative — it overestimates to avoid VRAM contention.

func PruneSplitModelDirs added in v0.34.1

func PruneSplitModelDirs(root string, before time.Time) error

PruneSplitModelDirs removes split-model alias directories older than before.

func SetupLlamaServerCommandEnv added in v0.30.0

func SetupLlamaServerCommandEnv(cmd *exec.Cmd, exe string, gpuLibs []string, extraEnvs map[string]string)

SetupLlamaServerCommandEnv configures the environment for a llama-server subprocess so discovery and real model runners use the same library search paths and GPU backend selection.

func ShouldRetryWithMetalTensorDisabled added in v0.23.0

func ShouldRetryWithMetalTensorDisabled(err error, status *StatusWriter) bool

ShouldRetryWithMetalTensorDisabled detects Metal tensor API initialization failures where ggml's probe can pass but real kernel compilation or context creation fails. Retrying with GGML_METAL_TENSOR_DISABLE=1 keeps discovery and later runner launches on the conservative Metal path.

Types

type ChatRequest added in v0.30.0

type ChatRequest struct {
	Messages []api.Message
	Tools    api.Tools
	Format   json.RawMessage
	Options  *api.Options
	Think    *api.ThinkValue
	Shift    bool

	Logprobs    bool
	TopLogprobs int
}

type ChatResponse added in v0.30.0

type ChatResponse struct {
	Message               api.Message   `json:"message"`
	DoneReason            DoneReason    `json:"done_reason"`
	Done                  bool          `json:"done"`
	PromptEvalCount       int           `json:"prompt_eval_count"`
	PromptEvalCachedCount *int          `json:"prompt_eval_cached_count,omitempty"`
	PromptEvalDuration    time.Duration `json:"prompt_eval_duration"`
	EvalCount             int           `json:"eval_count"`
	EvalDuration          time.Duration `json:"eval_duration"`
	Logprobs              []Logprob     `json:"logprobs,omitempty"`
}

type CompletionRequest added in v0.1.32

type CompletionRequest struct {
	Prompt  string
	Format  json.RawMessage
	Media   []MediaData
	Options *api.Options

	Shift           bool
	Truncate        bool
	PreservedTokens []string // parser tokens to render as text; ignored by non-llama-server runners
	ToolCallTag     string   // raw generic tool parser tag, if any
	LeadingBOS      string   // textual BOS emitted by Go rendering, if any
	// ThinkingClose holds the strings any of which ends the thinking the
	// response begins with, which Format leaves free; none when the response
	// starts in content.
	ThinkingClose []string

	// Logprobs specifies whether to include log probabilities in the response
	Logprobs bool

	// TopLogprobs specifies the number of most likely alternative tokens to return (0-20)
	TopLogprobs int
}

type CompletionResponse added in v0.1.32

type CompletionResponse struct {
	Content               string        `json:"content"`
	DoneReason            DoneReason    `json:"done_reason"`
	Done                  bool          `json:"done"`
	PromptEvalCount       int           `json:"prompt_eval_count"`
	PromptEvalCachedCount *int          `json:"prompt_eval_cached_count,omitempty"`
	PromptEvalDuration    time.Duration `json:"prompt_eval_duration"`
	EvalCount             int           `json:"eval_count"`
	EvalDuration          time.Duration `json:"eval_duration"`

	// Logprobs contains log probability information if requested
	Logprobs []Logprob `json:"logprobs,omitempty"`
}

type DoneReason added in v0.6.5

type DoneReason int

DoneReason represents the reason why a completion response is done

const (
	DoneReasonStop DoneReason = iota
	DoneReasonLength
	DoneReasonConnectionClosed
)

func (DoneReason) String added in v0.6.5

func (d DoneReason) String() string

type ExitStatus added in v0.30.0

type ExitStatus int

func ExitStatusFromError added in v0.30.0

func ExitStatusFromError(err error) ExitStatus

func (ExitStatus) Known added in v0.30.0

func (s ExitStatus) Known() bool

func (ExitStatus) LogValue added in v0.30.0

func (s ExitStatus) LogValue() slog.Value

func (ExitStatus) String added in v0.30.0

func (s ExitStatus) String() string

type LlamaServer added in v0.1.32

type LlamaServer interface {
	ModelPath() string
	Load(ctx context.Context, systemInfo ml.SystemInfo, gpus []ml.DeviceInfo, requireFull bool) ([]ml.DeviceID, error)
	Ping(ctx context.Context) error
	WaitUntilRunning(ctx context.Context) error
	Completion(ctx context.Context, req CompletionRequest, fn func(CompletionResponse)) error
	Chat(ctx context.Context, req ChatRequest, fn func(ChatResponse)) error
	ApplyChatTemplate(ctx context.Context, req ChatRequest) (string, error)
	Embedding(ctx context.Context, input string, media [][]byte) ([]float32, int, error)
	Tokenize(ctx context.Context, content string) ([]int, error)
	Detokenize(ctx context.Context, tokens []int) (string, error)
	Close() error
	MemorySize() (total, vram uint64)
	VRAMByGPU(id ml.DeviceID) uint64
	Pid() int
	GetPort() int
	GetDeviceInfos(ctx context.Context) []ml.DeviceInfo
	HasExited() bool
	ContextLength() int
}

func NewLlamaServer added in v0.1.32

func NewLlamaServer(systemInfo ml.SystemInfo, gpus []ml.DeviceInfo, modelPath string, f *gguf.Model, adapters, projectors []string, opts api.Options, numParallel int, config LlamaServerConfig) (LlamaServer, error)

NewLlamaServer creates a new llama-server runner for the given model. All GGUF models are served via the upstream llama-server subprocess.

func NewLlamaServerRunner added in v0.30.0

func NewLlamaServerRunner(
	gpus []ml.DeviceInfo,
	modelPath string,
	f *gguf.Model,
	adapters, projectors []string,
	opts api.Options,
	numParallel int,
	kvCacheType string,
	config LlamaServerConfig,
) (LlamaServer, error)

NewLlamaServerRunner creates a new llama-server runner that wraps the upstream llama-server binary.

type LlamaServerConfig added in v0.30.0

type LlamaServerConfig struct {
	DisableJinja         bool
	ContextShift         bool
	EnableMTP            bool
	ManifestDigest       string
	DraftModelPath       string
	DraftModelShardPaths []string
}

type Logprob added in v0.12.11

type Logprob struct {
	TokenLogprob
	TopLogprobs []TokenLogprob `json:"top_logprobs,omitempty"`
}

Logprob contains log probability information for a generated token.

type MediaData added in v0.30.0

type MediaData struct {
	Data []byte `json:"data"`
	ID   int    `json:"id"`
	Kind MediaKind
}

func NewMediaData added in v0.30.0

func NewMediaData(id int, data []byte) MediaData

type MediaKind added in v0.30.0

type MediaKind string
const (
	MediaKindUnknown MediaKind = ""
	MediaKindImage   MediaKind = "image"
	MediaKindAudio   MediaKind = "audio"
)

func DetectMediaKind added in v0.30.0

func DetectMediaKind(data []byte) MediaKind

type Message added in v0.30.0

type Message struct {
	Role       string
	Content    string
	Thinking   string
	Media      []MediaData
	ToolCalls  []api.ToolCall
	ToolName   string
	ToolCallID string
}

func MessageFromAPI added in v0.30.0

func MessageFromAPI(msg api.Message) Message

type NativeDecision added in v0.40.12

type NativeDecision struct {
	State     json.RawMessage  `json:"state,omitempty"`
	Questions []NativeQuestion `json:"questions"`
}

NativeDecision is a System One request as its caller wrote it.

type NativeQuestion added in v0.40.12

type NativeQuestion struct {
	Name     string          `json:"name"`
	Question json.RawMessage `json:"question"`
}

NativeQuestion keeps one question's JSON in request order.

type ScoreField added in v0.35.1

type ScoreField struct {
	Type     int      `json:"type"`
	Question [2]int   `json:"question"`
	Options  [][2]int `json:"options"`
}

ScoreField identifies a question and its allowed options for a decision head. Spans index ScoreRequest.Segments; the runner maps them to token offsets. Type is 0 for noul, 1 for choice, and 2 for score.

type ScorePointerRow added in v0.40.0

type ScorePointerRow struct {
	Prefix  string   `json:"prefix"`
	Prompt  string   `json:"prompt"`
	Type    int      `json:"type"`
	Options [][2]int `json:"options"`
}

ScorePointerRow is an independent state/question pair for a pointer head. Options are byte spans in Prompt; Type is 0 for noul, 1 for choice, 2 for score.

type ScoreQuestion added in v0.40.0

type ScoreQuestion struct {
	Type         string   `json:"type"`
	Instructions string   `json:"instructions"`
	Options      []string `json:"options"`
}

ScoreQuestion preserves a field's schema for model-native decision heads.

type ScoreRequest added in v0.35.0

type ScoreRequest struct {
	// System is the Modelfile instruction for scorers that assemble a native prompt.
	// Generic candidate rows already include it in their rendered prompts.
	System        string            `json:"system,omitempty"`
	PointerRows   []ScorePointerRow `json:"pointer_rows,omitempty"`
	State         string            `json:"state,omitempty"`
	Rows          []ScoreRow        `json:"rows,omitempty"`
	Segments      []string          `json:"segments,omitempty"`
	Fields        []ScoreField      `json:"fields,omitempty"`
	MaxTokens     int               `json:"max_tokens"`
	Images        []api.ImageData   `json:"images,omitempty"`
	ImagePosition int               `json:"image_position,omitempty"` // Segment boundary at which images are inserted.
	// Native carries a whole System One request to a model that renders and
	// answers it itself; Images are its media in order.
	Native *NativeDecision `json:"native,omitempty"`
}

ScoreRequest carries prompt rows or model-native decision head inputs.

type ScoreResponse added in v0.35.0

type ScoreResponse struct {
	// Logits may be log probabilities: a shared offset within a row does not
	// affect the softmax over its candidates.
	Logits       [][]float32 `json:"logits"`
	InputTokens  int         `json:"input_tokens"`            // Sum of complete prompt lengths, including shared tokens.
	OutputTokens int         `json:"output_tokens,omitempty"` // Tokens generated internally to obtain candidate scores.
	CachedTokens *int        `json:"cached_tokens,omitempty"` // Subset of InputTokens restored instead of evaluated; nil when unavailable.
	// Answers are complete System One answers, one per NativeDecision question.
	Answers []json.RawMessage `json:"answers,omitempty"`
}

type ScoreRow added in v0.35.0

type ScoreRow struct {
	Prompt     string         `json:"prompt"`
	Candidates []string       `json:"candidates"`
	Question   *ScoreQuestion `json:"question,omitempty"`
}

type Scorer added in v0.35.0

type Scorer interface {
	Score(context.Context, ScoreRequest) (ScoreResponse, error)
}

Scorer is an optional runner capability for bounded decision scoring. Inputs are already encoded; no chat template or generation options apply.

type ServerStatus added in v0.1.32

type ServerStatus int
const (
	ServerStatusReady ServerStatus = iota
	ServerStatusNoSlotsAvailable
	ServerStatusLaunched
	ServerStatusLoadingModel
	ServerStatusNotResponding
	ServerStatusError
)

func (ServerStatus) String added in v0.6.2

func (s ServerStatus) String() string

type ServerStatusResponse added in v0.6.2

type ServerStatusResponse struct {
	Status   ServerStatus `json:"status"`
	Progress float32      `json:"progress"`
}

type Speaker added in v0.40.9

type Speaker interface {
	Speech(ctx context.Context, req SpeechRequest, fn func(pcm []byte) error) error
}

Speaker is implemented by runners that serve speech models. Speech streams SpeechSampleRate mono 16-bit little-endian PCM to fn as it is generated.

type SpeechRequest added in v0.40.9

type SpeechRequest struct {
	Input string `json:"input"`
	Voice string `json:"voice,omitempty"`
	// Prompter maps the request onto llama-server's /tts; runners with their
	// own speech models ignore it.
	Prompter renderers.SpeechPrompter `json:"-"`
}

SpeechRequest asks a speech model to read Input aloud.

type StatusWriter added in v0.1.32

type StatusWriter struct {
	// contains filtered or unexported fields
}

StatusWriter is a writer that captures error messages from the llama runner process

func NewStatusWriter added in v0.1.32

func NewStatusWriter(out io.Writer) *StatusWriter

func (*StatusWriter) AppendError added in v0.23.0

func (w *StatusWriter) AppendError(msg string)

func (*StatusWriter) LastError added in v0.23.0

func (w *StatusWriter) LastError() string

func (*StatusWriter) SetLastError added in v0.23.0

func (w *StatusWriter) SetLastError(msg string)

func (*StatusWriter) Write added in v0.1.32

func (w *StatusWriter) Write(b []byte) (int, error)

type TokenLogprob added in v0.12.11

type TokenLogprob struct {
	Token   string  `json:"token"`
	Logprob float64 `json:"logprob"`
}

TokenLogprob represents log probability information for a single token alternative.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL