Documentation
¶
Overview ¶
llama_server.go wraps the llama-server binary as a subprocess
Ollama uses two chat paths with llama-server. Models with explicit Ollama renderers/parsers, Harmony handling, MLX, or an enabled Go TEMPLATE layer still render prompts in Go and call /completion. Other GGUF chat models use llama-server's chat_template handling through /v1/chat/completions.
For structured output, a JSON schema is passed to llama-server via its json_schema field and the "json" format as a builtin grammar via the grammar field. A format that applies after a response's thinking is sent as a grammar built around the GBNF llama-server itself derives from the schema.
llama-server auto-detects GPU layers (-ngl), thread count (-t), and flash attention (--flash-attn).
Index ¶
- Constants
- Variables
- func AddRunnerToJob(int) error
- func AudioFormat(data []byte) (string, bool)
- func FindLlamaServer() (string, error)
- func IsLlamaCPP(s LlamaServer) bool
- func IsOutOfMemory(err error) bool
- func IsOutOfMemoryMessage(msg string) bool
- func LlamaServerBatch(f *gguf.Model, opts api.Options, completion bool) (int, uint64)
- func LlamaServerBatchMemory(batch int) uint64
- func LlamaServerFlashAttention(gpus []ml.DeviceInfo) ml.FlashAttentionType
- func LoadModel(model string, maxArraySize int, shards ...string) (*gguf.Model, error)
- func PredictServerVRAM(modelPath string, f *gguf.Model, numCtx int) uint64
- func PruneSplitModelDirs(root string, before time.Time) error
- func SetupLlamaServerCommandEnv(cmd *exec.Cmd, exe string, gpuLibs []string, extraEnvs map[string]string)
- func ShouldRetryWithMetalTensorDisabled(err error, status *StatusWriter) bool
- type ChatRequest
- type ChatResponse
- type CompletionRequest
- type CompletionResponse
- type DoneReason
- type ExitStatus
- type LlamaServer
- type LlamaServerConfig
- type Logprob
- type MediaData
- type MediaKind
- type Message
- type NativeDecision
- type NativeQuestion
- type ScoreField
- type ScorePointerRow
- type ScoreQuestion
- type ScoreRequest
- type ScoreResponse
- type ScoreRow
- type Scorer
- type ServerStatus
- type ServerStatusResponse
- type Speaker
- type SpeechRequest
- type StatusWriter
- type TokenLogprob
Constants ¶
const (
DefaultEmbeddingNumBatch = 2048
)
DefaultEmbeddingNumBatch is the default NumBatch used for embedding models when neither the model nor the request specifies num_batch.
const SpeechSampleRate = 24000
SpeechSampleRate is the rate of the PCM a Speaker streams.
Variables ¶
var ErrLoadRequiredFull = errors.New("unable to load full model on GPU")
var LlamaServerSysProcAttr = &syscall.SysProcAttr{}
Functions ¶
func AddRunnerToJob ¶ added in v0.40.6
AddRunnerToJob is a no-op on platforms without Windows Job Objects.
func AudioFormat ¶ added in v0.30.0
func FindLlamaServer ¶ added in v0.30.0
FindLlamaServer locates the llama-server binary in lib/ollama/. There is a single binary that dynamically loads GPU backends at runtime.
func IsLlamaCPP ¶ added in v0.40.0
func IsLlamaCPP(s LlamaServer) bool
IsLlamaCPP reports whether the server is backed by the llama.cpp llama-server runner implementation.
func IsOutOfMemory ¶ added in v0.30.0
func IsOutOfMemoryMessage ¶ added in v0.30.0
func LlamaServerBatch ¶ added in v0.40.4
LlamaServerBatch resolves the batch size and extra memory allowance used when deciding whether to load a model alongside other models.
func LlamaServerBatchMemory ¶ added in v0.40.4
LlamaServerBatchMemory is the conservative extra memory allowance for a generation or whole-prompt decision batch above the default batch size.
func LlamaServerFlashAttention ¶ added in v0.30.0
func LlamaServerFlashAttention(gpus []ml.DeviceInfo) ml.FlashAttentionType
LlamaServerFlashAttention resolves the flash-attention mode passed to llama-server.
func LoadModel ¶ added in v0.1.33
LoadModel loads GGUF model metadata from disk.
It collects array values for arrays with a size less than or equal to maxArraySize. If maxArraySize is 0, the default value of 1024 is used. If the maxArraySize is negative, all arrays are collected.
func PredictServerVRAM ¶ added in v0.30.0
PredictServerVRAM estimates VRAM usage for a model without spawning llama-server. Uses model file size as a proxy for weights plus a rough KV cache estimate. This is intentionally conservative — it overestimates to avoid VRAM contention.
func PruneSplitModelDirs ¶ added in v0.34.1
PruneSplitModelDirs removes split-model alias directories older than before.
func SetupLlamaServerCommandEnv ¶ added in v0.30.0
func SetupLlamaServerCommandEnv(cmd *exec.Cmd, exe string, gpuLibs []string, extraEnvs map[string]string)
SetupLlamaServerCommandEnv configures the environment for a llama-server subprocess so discovery and real model runners use the same library search paths and GPU backend selection.
func ShouldRetryWithMetalTensorDisabled ¶ added in v0.23.0
func ShouldRetryWithMetalTensorDisabled(err error, status *StatusWriter) bool
ShouldRetryWithMetalTensorDisabled detects Metal tensor API initialization failures where ggml's probe can pass but real kernel compilation or context creation fails. Retrying with GGML_METAL_TENSOR_DISABLE=1 keeps discovery and later runner launches on the conservative Metal path.
Types ¶
type ChatRequest ¶ added in v0.30.0
type ChatResponse ¶ added in v0.30.0
type ChatResponse struct {
Message api.Message `json:"message"`
DoneReason DoneReason `json:"done_reason"`
Done bool `json:"done"`
PromptEvalCount int `json:"prompt_eval_count"`
PromptEvalCachedCount *int `json:"prompt_eval_cached_count,omitempty"`
PromptEvalDuration time.Duration `json:"prompt_eval_duration"`
EvalCount int `json:"eval_count"`
EvalDuration time.Duration `json:"eval_duration"`
Logprobs []Logprob `json:"logprobs,omitempty"`
}
type CompletionRequest ¶ added in v0.1.32
type CompletionRequest struct {
Prompt string
Format json.RawMessage
Media []MediaData
Options *api.Options
Shift bool
Truncate bool
PreservedTokens []string // parser tokens to render as text; ignored by non-llama-server runners
ToolCallTag string // raw generic tool parser tag, if any
LeadingBOS string // textual BOS emitted by Go rendering, if any
// ThinkingClose holds the strings any of which ends the thinking the
// response begins with, which Format leaves free; none when the response
// starts in content.
ThinkingClose []string
// Logprobs specifies whether to include log probabilities in the response
Logprobs bool
// TopLogprobs specifies the number of most likely alternative tokens to return (0-20)
TopLogprobs int
}
type CompletionResponse ¶ added in v0.1.32
type CompletionResponse struct {
Content string `json:"content"`
DoneReason DoneReason `json:"done_reason"`
Done bool `json:"done"`
PromptEvalCount int `json:"prompt_eval_count"`
PromptEvalCachedCount *int `json:"prompt_eval_cached_count,omitempty"`
PromptEvalDuration time.Duration `json:"prompt_eval_duration"`
EvalCount int `json:"eval_count"`
EvalDuration time.Duration `json:"eval_duration"`
// Logprobs contains log probability information if requested
Logprobs []Logprob `json:"logprobs,omitempty"`
}
type DoneReason ¶ added in v0.6.5
type DoneReason int
DoneReason represents the reason why a completion response is done
const ( DoneReasonStop DoneReason = iota DoneReasonLength DoneReasonConnectionClosed )
func (DoneReason) String ¶ added in v0.6.5
func (d DoneReason) String() string
type ExitStatus ¶ added in v0.30.0
type ExitStatus int
func ExitStatusFromError ¶ added in v0.30.0
func ExitStatusFromError(err error) ExitStatus
func (ExitStatus) Known ¶ added in v0.30.0
func (s ExitStatus) Known() bool
func (ExitStatus) LogValue ¶ added in v0.30.0
func (s ExitStatus) LogValue() slog.Value
func (ExitStatus) String ¶ added in v0.30.0
func (s ExitStatus) String() string
type LlamaServer ¶ added in v0.1.32
type LlamaServer interface {
ModelPath() string
Load(ctx context.Context, systemInfo ml.SystemInfo, gpus []ml.DeviceInfo, requireFull bool) ([]ml.DeviceID, error)
Ping(ctx context.Context) error
WaitUntilRunning(ctx context.Context) error
Completion(ctx context.Context, req CompletionRequest, fn func(CompletionResponse)) error
Chat(ctx context.Context, req ChatRequest, fn func(ChatResponse)) error
ApplyChatTemplate(ctx context.Context, req ChatRequest) (string, error)
Embedding(ctx context.Context, input string, media [][]byte) ([]float32, int, error)
Tokenize(ctx context.Context, content string) ([]int, error)
Detokenize(ctx context.Context, tokens []int) (string, error)
Close() error
MemorySize() (total, vram uint64)
VRAMByGPU(id ml.DeviceID) uint64
Pid() int
GetPort() int
GetDeviceInfos(ctx context.Context) []ml.DeviceInfo
HasExited() bool
ContextLength() int
}
func NewLlamaServer ¶ added in v0.1.32
func NewLlamaServer(systemInfo ml.SystemInfo, gpus []ml.DeviceInfo, modelPath string, f *gguf.Model, adapters, projectors []string, opts api.Options, numParallel int, config LlamaServerConfig) (LlamaServer, error)
NewLlamaServer creates a new llama-server runner for the given model. All GGUF models are served via the upstream llama-server subprocess.
func NewLlamaServerRunner ¶ added in v0.30.0
func NewLlamaServerRunner( gpus []ml.DeviceInfo, modelPath string, f *gguf.Model, adapters, projectors []string, opts api.Options, numParallel int, kvCacheType string, config LlamaServerConfig, ) (LlamaServer, error)
NewLlamaServerRunner creates a new llama-server runner that wraps the upstream llama-server binary.
type LlamaServerConfig ¶ added in v0.30.0
type Logprob ¶ added in v0.12.11
type Logprob struct {
TokenLogprob
TopLogprobs []TokenLogprob `json:"top_logprobs,omitempty"`
}
Logprob contains log probability information for a generated token.
type MediaData ¶ added in v0.30.0
func NewMediaData ¶ added in v0.30.0
type Message ¶ added in v0.30.0
type Message struct {
Role string
Content string
Thinking string
Media []MediaData
ToolCalls []api.ToolCall
ToolName string
ToolCallID string
}
func MessageFromAPI ¶ added in v0.30.0
type NativeDecision ¶ added in v0.40.12
type NativeDecision struct {
State json.RawMessage `json:"state,omitempty"`
Questions []NativeQuestion `json:"questions"`
}
NativeDecision is a System One request as its caller wrote it.
type NativeQuestion ¶ added in v0.40.12
type NativeQuestion struct {
Name string `json:"name"`
Question json.RawMessage `json:"question"`
}
NativeQuestion keeps one question's JSON in request order.
type ScoreField ¶ added in v0.35.1
type ScoreField struct {
Type int `json:"type"`
Question [2]int `json:"question"`
Options [][2]int `json:"options"`
}
ScoreField identifies a question and its allowed options for a decision head. Spans index ScoreRequest.Segments; the runner maps them to token offsets. Type is 0 for noul, 1 for choice, and 2 for score.
type ScorePointerRow ¶ added in v0.40.0
type ScorePointerRow struct {
Prefix string `json:"prefix"`
Prompt string `json:"prompt"`
Type int `json:"type"`
Options [][2]int `json:"options"`
}
ScorePointerRow is an independent state/question pair for a pointer head. Options are byte spans in Prompt; Type is 0 for noul, 1 for choice, 2 for score.
type ScoreQuestion ¶ added in v0.40.0
type ScoreQuestion struct {
Type string `json:"type"`
Instructions string `json:"instructions"`
Options []string `json:"options"`
}
ScoreQuestion preserves a field's schema for model-native decision heads.
type ScoreRequest ¶ added in v0.35.0
type ScoreRequest struct {
// System is the Modelfile instruction for scorers that assemble a native prompt.
// Generic candidate rows already include it in their rendered prompts.
System string `json:"system,omitempty"`
PointerRows []ScorePointerRow `json:"pointer_rows,omitempty"`
State string `json:"state,omitempty"`
Rows []ScoreRow `json:"rows,omitempty"`
Segments []string `json:"segments,omitempty"`
Fields []ScoreField `json:"fields,omitempty"`
MaxTokens int `json:"max_tokens"`
Images []api.ImageData `json:"images,omitempty"`
ImagePosition int `json:"image_position,omitempty"` // Segment boundary at which images are inserted.
// Native carries a whole System One request to a model that renders and
// answers it itself; Images are its media in order.
Native *NativeDecision `json:"native,omitempty"`
}
ScoreRequest carries prompt rows or model-native decision head inputs.
type ScoreResponse ¶ added in v0.35.0
type ScoreResponse struct {
// Logits may be log probabilities: a shared offset within a row does not
// affect the softmax over its candidates.
Logits [][]float32 `json:"logits"`
InputTokens int `json:"input_tokens"` // Sum of complete prompt lengths, including shared tokens.
OutputTokens int `json:"output_tokens,omitempty"` // Tokens generated internally to obtain candidate scores.
CachedTokens *int `json:"cached_tokens,omitempty"` // Subset of InputTokens restored instead of evaluated; nil when unavailable.
// Answers are complete System One answers, one per NativeDecision question.
Answers []json.RawMessage `json:"answers,omitempty"`
}
type ScoreRow ¶ added in v0.35.0
type ScoreRow struct {
Prompt string `json:"prompt"`
Candidates []string `json:"candidates"`
Question *ScoreQuestion `json:"question,omitempty"`
}
type Scorer ¶ added in v0.35.0
type Scorer interface {
Score(context.Context, ScoreRequest) (ScoreResponse, error)
}
Scorer is an optional runner capability for bounded decision scoring. Inputs are already encoded; no chat template or generation options apply.
type ServerStatus ¶ added in v0.1.32
type ServerStatus int
const ( ServerStatusReady ServerStatus = iota ServerStatusNoSlotsAvailable ServerStatusLaunched ServerStatusLoadingModel ServerStatusNotResponding ServerStatusError )
func (ServerStatus) String ¶ added in v0.6.2
func (s ServerStatus) String() string
type ServerStatusResponse ¶ added in v0.6.2
type ServerStatusResponse struct {
Status ServerStatus `json:"status"`
Progress float32 `json:"progress"`
}
type Speaker ¶ added in v0.40.9
type Speaker interface {
Speech(ctx context.Context, req SpeechRequest, fn func(pcm []byte) error) error
}
Speaker is implemented by runners that serve speech models. Speech streams SpeechSampleRate mono 16-bit little-endian PCM to fn as it is generated.
type SpeechRequest ¶ added in v0.40.9
type SpeechRequest struct {
Input string `json:"input"`
Voice string `json:"voice,omitempty"`
// Prompter maps the request onto llama-server's /tts; runners with their
// own speech models ignore it.
Prompter renderers.SpeechPrompter `json:"-"`
}
SpeechRequest asks a speech model to read Input aloud.
type StatusWriter ¶ added in v0.1.32
type StatusWriter struct {
// contains filtered or unexported fields
}
StatusWriter is a writer that captures error messages from the llama runner process
func NewStatusWriter ¶ added in v0.1.32
func NewStatusWriter(out io.Writer) *StatusWriter
func (*StatusWriter) AppendError ¶ added in v0.23.0
func (w *StatusWriter) AppendError(msg string)
func (*StatusWriter) LastError ¶ added in v0.23.0
func (w *StatusWriter) LastError() string
func (*StatusWriter) SetLastError ¶ added in v0.23.0
func (w *StatusWriter) SetLastError(msg string)
type TokenLogprob ¶ added in v0.12.11
TokenLogprob represents log probability information for a single token alternative.