Documentation
¶
Index ¶
- Variables
- func CheckRuntime() error
- func EmbedCapacityBudget() int64
- func Execute(args []string) error
- func PredictedEmbedAllocationBytes(tokens int, hidden int32) int64
- func SupportsArchitecture(arch string) bool
- func SupportsDraftArchitecture(arch string) bool
- type Client
- func (c *Client) ApplyChatTemplate(ctx context.Context, req llm.ChatRequest) (string, error)
- func (c *Client) Chat(ctx context.Context, req llm.ChatRequest, fn func(llm.ChatResponse)) error
- func (c *Client) Close() error
- func (c *Client) Completion(ctx context.Context, req llm.CompletionRequest, ...) error
- func (c *Client) ContextLength() int
- func (c *Client) Detokenize(ctx context.Context, tokens []int) (string, error)
- func (c *Client) Embedding(ctx context.Context, input string, media [][]byte) ([]float32, int, error)
- func (c *Client) EmbeddingDimensions() []int
- func (c *Client) GetDeviceInfos(ctx context.Context) []ml.DeviceInfo
- func (c *Client) GetPort() int
- func (c *Client) HasExited() bool
- func (c *Client) Load(ctx context.Context, systemInfo ml.SystemInfo, gpus []ml.DeviceInfo, ...) ([]ml.DeviceID, error)
- func (c *Client) MemorySize() (total, vram uint64)
- func (c *Client) ModelPath() string
- func (c *Client) Pid() int
- func (c *Client) Ping(ctx context.Context) error
- func (c *Client) Score(ctx context.Context, input llm.ScoreRequest) (llm.ScoreResponse, error)
- func (c *Client) Tokenize(ctx context.Context, content string) ([]int, error)
- func (c *Client) VRAMByGPU(id ml.DeviceID) uint64
- func (c *Client) WaitUntilRunning(ctx context.Context) error
- type CompletionRequest
- type CompletionResponse
- type EmbedResponse
- type EmbeddingRequest
- type Request
- type Runner
Constants ¶
This section is empty.
Variables ¶
Functions ¶
func CheckRuntime ¶
func CheckRuntime() error
CheckRuntime reports whether an MLX dynamic library was loaded successfully.
func EmbedCapacityBudget ¶ added in v0.40.0
func EmbedCapacityBudget() int64
EmbedCapacityBudget is the byte budget a single embed request's predicted allocations must fit within.
func PredictedEmbedAllocationBytes ¶ added in v0.40.0
PredictedEmbedAllocationBytes estimates the forward's dominant allocations: the attention mask (L*L float32, worst case), the hidden output [1, L, D] float32, and the token row.
func SupportsArchitecture ¶
SupportsArchitecture reports whether the MLX runner has a constructor for arch.
func SupportsDraftArchitecture ¶
SupportsDraftArchitecture reports whether the MLX runner has a draft constructor for arch.
Types ¶
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client wraps an MLX runner subprocess to implement llm.LlamaServer for LLM models.
func NewClient ¶
NewClient prepares a new MLX runner client for LLM models. The subprocess is not started until Load() is called.
func (*Client) ApplyChatTemplate ¶
func (*Client) Chat ¶
func (c *Client) Chat(ctx context.Context, req llm.ChatRequest, fn func(llm.ChatResponse)) error
func (*Client) Completion ¶
func (c *Client) Completion(ctx context.Context, req llm.CompletionRequest, fn func(llm.CompletionResponse)) error
Completion implements llm.LlamaServer.
func (*Client) ContextLength ¶
func (*Client) Detokenize ¶
Detokenize implements llm.LlamaServer.
func (*Client) EmbeddingDimensions ¶ added in v0.40.0
EmbeddingDimensions returns the runner's declared matryoshka set, cached from the last successful status response. Empty when the runner doesn't advertise one.
func (*Client) GetDeviceInfos ¶
func (c *Client) GetDeviceInfos(ctx context.Context) []ml.DeviceInfo
GetDeviceInfos implements llm.LlamaServer.
func (*Client) Load ¶
func (c *Client) Load(ctx context.Context, systemInfo ml.SystemInfo, gpus []ml.DeviceInfo, requireFull bool) ([]ml.DeviceID, error)
Load checks whether the model fits in GPU memory and starts the subprocess.
func (*Client) MemorySize ¶
MemorySize implements llm.LlamaServer.
func (*Client) Score ¶ added in v0.40.0
func (c *Client) Score(ctx context.Context, input llm.ScoreRequest) (llm.ScoreResponse, error)
type CompletionRequest ¶
type CompletionResponse ¶
type EmbedResponse ¶ added in v0.40.0
type EmbedResponse struct {
Embedding []float32
PromptEvalCount int
Error *api.StatusError
}
EmbedResponse carries one embedding, its prompt token count, or an error.
type EmbeddingRequest ¶ added in v0.40.0
type EmbeddingRequest struct {
Content string
Dimensions int
Media [][]byte
EmbedResponses chan EmbedResponse
Ctx context.Context //nolint:containedctx // Queued requests carry caller cancellation to the runner.
}
EmbeddingRequest is a short-lived embedding request carried on the runner's EmbedRequests channel from the HTTP handler to the runner goroutine (same lifecycle shape as completions' Request/Requests).
type Request ¶
type Request struct {
CompletionRequest
Responses chan CompletionResponse
Pipeline func(context.Context, Request) error
Ctx context.Context //nolint:containedctx // Queued requests carry caller cancellation to the runner.
Tokens []int32
MediaItems []mediaItem
Layout any // opaque PrepareMedia layout state, stamped on every batch
SamplerOpts sample.Options
Grammar *grammarCompilation
}
Request is a short-lived struct that carries a completion request through a channel from the HTTP handler to the runner goroutine. The ctx field must travel with the request so that cancellation propagates across the channel boundary.
type Runner ¶
type Runner struct {
Model model.Model
Tokenizer *tokenizer.Tokenizer
Requests chan Request
EmbedRequests chan EmbeddingRequest
Sampler *sample.Sampler
// contains filtered or unexported fields
}
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
clef
Package clef implements Cloudflare's Qwen3.5 joint-schema decision model.
|
Package clef implements Cloudflare's Qwen3.5 joint-schema decision model. |
|
cohere2_moe
Package cohere2_moe provides the Cohere2 MoE (Command A family, North) text model implementation for MLX.
|
Package cohere2_moe provides the Cohere2 MoE (Command A family, North) text model implementation for MLX. |
|
dflash
Package dflash implements the DFlash block-diffusion draft model: qwen3-shaped layers drafting a whole block per forward, conditioned on tapped target hidden states as key/value context.
|
Package dflash implements the DFlash block-diffusion draft model: qwen3-shaped layers drafting a whole block per forward, conditioned on tapped target hidden states as key/value context. |
|
gemma4
Package gemma4 provides the Gemma 4 model implementation for MLX.
|
Package gemma4 provides the Gemma 4 model implementation for MLX. |
|
gemma4embedding
Package gemma4embedding implements the Gemma4Embedding (EmbeddingGemma v2) text embedding model for MLX.
|
Package gemma4embedding implements the Gemma4Embedding (EmbeddingGemma v2) text embedding model for MLX. |
|
glimmer
Package glimmer provides the Meta Glimmer text model implementation for MLX.
|
Package glimmer provides the Meta Glimmer text model implementation for MLX. |
|
glm4_moe_lite
Package glm4_moe_lite provides the GLM4-MoE-Lite implementation for MLX.
|
Package glm4_moe_lite provides the GLM4-MoE-Lite implementation for MLX. |
|
kev
Package kev implements the Qwen3.5/Qwen3.8 Kev decision head.
|
Package kev implements the Qwen3.5/Qwen3.8 Kev decision head. |
|
laguna
Package laguna provides the Poolside Laguna text model implementation for MLX.
|
Package laguna provides the Poolside Laguna text model implementation for MLX. |
|
laya
Package laya implements the ModernBERT decision model from https://github.com/NandhaKishorM/laya (Apache-2.0).
|
Package laya implements the ModernBERT decision model from https://github.com/NandhaKishorM/laya (Apache-2.0). |
|
llama
Package llama provides a Llama-style decoder-only transformer for MLX.
|
Package llama provides a Llama-style decoder-only transformer for MLX. |
|
nemotron_h
Package nemotron_h provides the Nemotron-H text model implementation for MLX.
|
Package nemotron_h provides the Nemotron-H text model implementation for MLX. |
|
qwen3
Package qwen3 provides the Qwen3 text model implementation for MLX.
|
Package qwen3 provides the Qwen3 text model implementation for MLX. |
|
qwen3_5
Package qwen3_5 provides the Qwen 3.5 text and MoE implementation for MLX.
|
Package qwen3_5 provides the Qwen 3.5 text and MoE implementation for MLX. |
|
qwen3_5_moe
Package qwen3_5_moe registers Qwen 3.5 MoE architecture aliases.
|
Package qwen3_5_moe registers Qwen 3.5 MoE architecture aliases. |
|
qwen4_exp
Package qwen4_exp implements the Qwen 4 MLX model family.
|
Package qwen4_exp implements the Qwen 4 MLX model family. |
|
strands
Package strands implements Strands Decider's Qwen3.5 pointer-head model.
|
Package strands implements Strands Decider's Qwen3.5 pointer-head model. |