Documentation
¶
Index ¶
- Variables
- func CheckRuntime() error
- func Execute(args []string) error
- func SupportsArchitecture(arch string) bool
- func SupportsDraftArchitecture(arch string) bool
- type Client
- func (c *Client) ApplyChatTemplate(ctx context.Context, req llm.ChatRequest) (string, error)
- func (c *Client) Chat(ctx context.Context, req llm.ChatRequest, fn func(llm.ChatResponse)) error
- func (c *Client) Close() error
- func (c *Client) Completion(ctx context.Context, req llm.CompletionRequest, ...) error
- func (c *Client) ContextLength() int
- func (c *Client) Detokenize(ctx context.Context, tokens []int) (string, error)
- func (c *Client) Embedding(ctx context.Context, input string) ([]float32, int, error)
- func (c *Client) GetDeviceInfos(ctx context.Context) []ml.DeviceInfo
- func (c *Client) GetPort() int
- func (c *Client) HasExited() bool
- func (c *Client) Load(ctx context.Context, systemInfo ml.SystemInfo, gpus []ml.DeviceInfo, ...) ([]ml.DeviceID, error)
- func (c *Client) MemorySize() (total, vram uint64)
- func (c *Client) ModelPath() string
- func (c *Client) Pid() int
- func (c *Client) Ping(ctx context.Context) error
- func (c *Client) Tokenize(ctx context.Context, content string) ([]int, error)
- func (c *Client) VRAMByGPU(id ml.DeviceID) uint64
- func (c *Client) WaitUntilRunning(ctx context.Context) error
- type CompletionRequest
- type CompletionResponse
- type Request
- type Runner
Constants ¶
This section is empty.
Variables ¶
Functions ¶
func CheckRuntime ¶
func CheckRuntime() error
CheckRuntime reports whether an MLX dynamic library was loaded successfully.
func SupportsArchitecture ¶
SupportsArchitecture reports whether the MLX runner has a constructor for arch.
func SupportsDraftArchitecture ¶
SupportsDraftArchitecture reports whether the MLX runner has a draft constructor for arch.
Types ¶
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client wraps an MLX runner subprocess to implement llm.LlamaServer for LLM models.
func NewClient ¶
NewClient prepares a new MLX runner client for LLM models. The subprocess is not started until Load() is called.
func (*Client) ApplyChatTemplate ¶
func (*Client) Chat ¶
func (c *Client) Chat(ctx context.Context, req llm.ChatRequest, fn func(llm.ChatResponse)) error
func (*Client) Completion ¶
func (c *Client) Completion(ctx context.Context, req llm.CompletionRequest, fn func(llm.CompletionResponse)) error
Completion implements llm.LlamaServer.
func (*Client) ContextLength ¶
func (*Client) Detokenize ¶
Detokenize implements llm.LlamaServer.
func (*Client) GetDeviceInfos ¶
func (c *Client) GetDeviceInfos(ctx context.Context) []ml.DeviceInfo
GetDeviceInfos implements llm.LlamaServer.
func (*Client) Load ¶
func (c *Client) Load(ctx context.Context, systemInfo ml.SystemInfo, gpus []ml.DeviceInfo, requireFull bool) ([]ml.DeviceID, error)
Load checks whether the model fits in GPU memory and starts the subprocess.
func (*Client) MemorySize ¶
MemorySize implements llm.LlamaServer.
type CompletionRequest ¶
type CompletionResponse ¶
type Request ¶
type Request struct {
CompletionRequest
Responses chan CompletionResponse
Pipeline func(context.Context, Request) error
Ctx context.Context //nolint:containedctx // Queued requests carry caller cancellation to the runner.
Tokens []int32
MediaItems []mediaItem
Layout any // opaque PrepareMedia layout state, stamped on every batch
SamplerOpts sample.Options
Grammar *grammarCompilation
}
Request is a short-lived struct that carries a completion request through a channel from the HTTP handler to the runner goroutine. The ctx field must travel with the request so that cancellation propagates across the channel boundary.
type Runner ¶
type Runner struct {
Model model.Model
Tokenizer *tokenizer.Tokenizer
Requests chan Request
Sampler *sample.Sampler
// contains filtered or unexported fields
}
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
cohere2_moe
Package cohere2_moe provides the Cohere2 MoE (Command A family, North) text model implementation for MLX.
|
Package cohere2_moe provides the Cohere2 MoE (Command A family, North) text model implementation for MLX. |
|
dflash
Package dflash implements the DFlash block-diffusion draft model: qwen3-shaped layers drafting a whole block per forward, conditioned on tapped target hidden states as key/value context.
|
Package dflash implements the DFlash block-diffusion draft model: qwen3-shaped layers drafting a whole block per forward, conditioned on tapped target hidden states as key/value context. |
|
gemma4
Package gemma4 provides the Gemma 4 model implementation for MLX.
|
Package gemma4 provides the Gemma 4 model implementation for MLX. |
|
glimmer
Package glimmer provides the Meta Glimmer text model implementation for MLX.
|
Package glimmer provides the Meta Glimmer text model implementation for MLX. |
|
glm4_moe_lite
Package glm4_moe_lite provides the GLM4-MoE-Lite implementation for MLX.
|
Package glm4_moe_lite provides the GLM4-MoE-Lite implementation for MLX. |
|
laguna
Package laguna provides the Poolside Laguna text model implementation for MLX.
|
Package laguna provides the Poolside Laguna text model implementation for MLX. |
|
llama
Package llama provides a Llama-style decoder-only transformer for MLX.
|
Package llama provides a Llama-style decoder-only transformer for MLX. |
|
nemotron_h
Package nemotron_h provides the Nemotron-H text model implementation for MLX.
|
Package nemotron_h provides the Nemotron-H text model implementation for MLX. |
|
qwen3
Package qwen3 provides the Qwen3 text model implementation for MLX.
|
Package qwen3 provides the Qwen3 text model implementation for MLX. |
|
qwen3_5
Package qwen3_5 provides the Qwen 3.5 text and MoE implementation for MLX.
|
Package qwen3_5 provides the Qwen 3.5 text and MoE implementation for MLX. |
|
qwen3_5_moe
Package qwen3_5_moe registers Qwen 3.5 MoE architecture aliases.
|
Package qwen3_5_moe registers Qwen 3.5 MoE architecture aliases. |
|
qwen4_exp
Package qwen4_exp implements the Qwen 4 MLX model family.
|
Package qwen4_exp implements the Qwen 4 MLX model family. |