mlxrunner

package
v0.40.13 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 28, 2026 License: MIT Imports: 50 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

View Source
var ErrRuntimeUnavailable = errors.New("MLX runtime is not available")

Functions

func CheckRuntime

func CheckRuntime() error

CheckRuntime reports whether an MLX dynamic library was loaded successfully.

func EmbedCapacityBudget added in v0.40.0

func EmbedCapacityBudget() int64

EmbedCapacityBudget is the byte budget a single embed request's predicted allocations must fit within.

func Execute

func Execute(args []string) error

func PredictedEmbedAllocationBytes added in v0.40.0

func PredictedEmbedAllocationBytes(tokens int, hidden int32) int64

PredictedEmbedAllocationBytes estimates the forward's dominant allocations: the attention mask (L*L float32, worst case), the hidden output [1, L, D] float32, and the token row.

func SupportsArchitecture

func SupportsArchitecture(arch string) bool

SupportsArchitecture reports whether the MLX runner has a constructor for arch.

func SupportsDraftArchitecture

func SupportsDraftArchitecture(arch string) bool

SupportsDraftArchitecture reports whether the MLX runner has a draft constructor for arch.

Types

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client wraps an MLX runner subprocess to implement llm.LlamaServer for LLM models.

func NewClient

func NewClient(modelName string, softContextLength int) (*Client, error)

NewClient prepares a new MLX runner client for LLM models. The subprocess is not started until Load() is called.

func (*Client) ApplyChatTemplate

func (c *Client) ApplyChatTemplate(ctx context.Context, req llm.ChatRequest) (string, error)

func (*Client) Chat

func (c *Client) Chat(ctx context.Context, req llm.ChatRequest, fn func(llm.ChatResponse)) error

func (*Client) Close

func (c *Client) Close() error

Close terminates the subprocess.

func (*Client) Completion

func (c *Client) Completion(ctx context.Context, req llm.CompletionRequest, fn func(llm.CompletionResponse)) error

Completion implements llm.LlamaServer.

func (*Client) ContextLength

func (c *Client) ContextLength() int

func (*Client) Detokenize

func (c *Client) Detokenize(ctx context.Context, tokens []int) (string, error)

Detokenize implements llm.LlamaServer.

func (*Client) Embedding

func (c *Client) Embedding(ctx context.Context, input string, media [][]byte) ([]float32, int, error)

func (*Client) EmbeddingDimensions added in v0.40.0

func (c *Client) EmbeddingDimensions() []int

EmbeddingDimensions returns the runner's declared matryoshka set, cached from the last successful status response. Empty when the runner doesn't advertise one.

func (*Client) GetDeviceInfos

func (c *Client) GetDeviceInfos(ctx context.Context) []ml.DeviceInfo

GetDeviceInfos implements llm.LlamaServer.

func (*Client) GetPort

func (c *Client) GetPort() int

GetPort implements llm.LlamaServer.

func (*Client) HasExited

func (c *Client) HasExited() bool

HasExited implements llm.LlamaServer.

func (*Client) Load

func (c *Client) Load(ctx context.Context, systemInfo ml.SystemInfo, gpus []ml.DeviceInfo, requireFull bool) ([]ml.DeviceID, error)

Load checks whether the model fits in GPU memory and starts the subprocess.

func (*Client) MemorySize

func (c *Client) MemorySize() (total, vram uint64)

MemorySize implements llm.LlamaServer.

func (*Client) ModelPath

func (c *Client) ModelPath() string

ModelPath implements llm.LlamaServer.

func (*Client) Pid

func (c *Client) Pid() int

Pid implements llm.LlamaServer.

func (*Client) Ping

func (c *Client) Ping(ctx context.Context) error

Ping implements llm.LlamaServer.

func (*Client) Score added in v0.40.0

func (c *Client) Score(ctx context.Context, input llm.ScoreRequest) (llm.ScoreResponse, error)

func (*Client) Speech added in v0.40.9

func (c *Client) Speech(ctx context.Context, input llm.SpeechRequest, fn func([]byte) error) error

func (*Client) Tokenize

func (c *Client) Tokenize(ctx context.Context, content string) ([]int, error)

Tokenize implements llm.LlamaServer.

func (*Client) VRAMByGPU

func (c *Client) VRAMByGPU(id ml.DeviceID) uint64

VRAMByGPU implements llm.LlamaServer.

func (*Client) WaitUntilRunning

func (c *Client) WaitUntilRunning(ctx context.Context) error

WaitUntilRunning waits for the subprocess to be ready.

type CompletionRequest

type CompletionRequest struct {
	Prompt      string
	Media       []llm.MediaData
	Format      json.RawMessage
	Options     api.Options
	Logprobs    bool
	TopLogprobs int
}

type CompletionResponse

type CompletionResponse struct {
	Content    string
	Done       bool
	DoneReason int

	PromptEvalCount       int
	PromptEvalCachedCount *int
	PromptEvalDuration    time.Duration
	EvalCount             int
	EvalDuration          time.Duration

	Logprobs []llm.Logprob

	Error *api.StatusError
}

type EmbedResponse added in v0.40.0

type EmbedResponse struct {
	Embedding       []float32
	PromptEvalCount int
	Error           *api.StatusError
}

EmbedResponse carries one embedding, its prompt token count, or an error.

type EmbeddingRequest added in v0.40.0

type EmbeddingRequest struct {
	Content        string
	Dimensions     int
	Media          [][]byte
	EmbedResponses chan EmbedResponse
	Ctx            context.Context //nolint:containedctx // Queued requests carry caller cancellation to the runner.
}

EmbeddingRequest is a short-lived embedding request carried on the runner's EmbedRequests channel from the HTTP handler to the runner goroutine (same lifecycle shape as completions' Request/Requests).

type Request

type Request struct {
	CompletionRequest
	Responses chan CompletionResponse
	Pipeline  func(context.Context, Request) error

	Ctx         context.Context //nolint:containedctx // Queued requests carry caller cancellation to the runner.
	Tokens      []int32
	MediaItems  []mediaItem
	Layout      any // opaque PrepareMedia layout state, stamped on every batch
	SamplerOpts sample.Options
	Grammar     *grammarCompilation
}

Request is a short-lived struct that carries a completion request through a channel from the HTTP handler to the runner goroutine. The ctx field must travel with the request so that cancellation propagates across the channel boundary.

type Runner

type Runner struct {
	Model model.Model

	Tokenizer     *tokenizer.Tokenizer
	Requests      chan Request
	EmbedRequests chan EmbeddingRequest
	Sampler       *sample.Sampler
	// contains filtered or unexported fields
}

func (*Runner) Close

func (r *Runner) Close()

func (*Runner) Load

func (r *Runner) Load(modelName string) error

func (*Runner) Prepare

func (r *Runner) Prepare(request *Request) (err error)

Prepare tokenizes the prompt and validates it against the model's context length. It is safe to call from any goroutine. On success it populates request.Tokens and adjusts request.Options.NumPredict.

func (*Runner) Run

func (r *Runner) Run(host, port string, mux http.Handler) error

func (*Runner) TextGenerationPipeline

func (r *Runner) TextGenerationPipeline(ctx context.Context, request Request) (err error)

Directories

Path Synopsis
clef
Package clef implements Cloudflare's Qwen3.5 joint-schema decision model.
Package clef implements Cloudflare's Qwen3.5 joint-schema decision model.
cohere2_moe
Package cohere2_moe provides the Cohere2 MoE (Command A family, North) text model implementation for MLX.
Package cohere2_moe provides the Cohere2 MoE (Command A family, North) text model implementation for MLX.
d1
Package d1 serves Liquid AI's d1 decision checkpoints, which answer System One questions from one non-causal forward pass over an LFM3-VL or LFM3-Omni backbone.
Package d1 serves Liquid AI's d1 decision checkpoints, which answer System One questions from one non-causal forward pass over an LFM3-VL or LFM3-Omni backbone.
d3
Package d3 implements the original Decision 3 code-readout checkpoints.
Package d3 implements the original Decision 3 code-readout checkpoints.
decisionqwen
Package decisionqwen routes Qwen3.5 decision checkpoints by their publisher-defined scoring format, since config.json names only the backbone.
Package decisionqwen routes Qwen3.5 decision checkpoints by their publisher-defined scoring format, since config.json names only the backbone.
dflash
Package dflash implements the DFlash block-diffusion draft model: qwen3-shaped layers drafting a whole block per forward, conditioned on tapped target hidden states as key/value context.
Package dflash implements the DFlash block-diffusion draft model: qwen3-shaped layers drafting a whole block per forward, conditioned on tapped target hidden states as key/value context.
flint
Package flint implements the Flint text model on MLX.
Package flint implements the Flint text model on MLX.
gemma4
Package gemma4 provides the Gemma 4 model implementation for MLX.
Package gemma4 provides the Gemma 4 model implementation for MLX.
gemma4embedding
Package gemma4embedding implements the Gemma4Embedding (EmbeddingGemma v2) text embedding model for MLX.
Package gemma4embedding implements the Gemma4Embedding (EmbeddingGemma v2) text embedding model for MLX.
glimmer
Package glimmer provides the Meta Glimmer text model implementation for MLX.
Package glimmer provides the Meta Glimmer text model implementation for MLX.
glm4_moe_lite
Package glm4_moe_lite provides the GLM4-MoE-Lite implementation for MLX.
Package glm4_moe_lite provides the GLM4-MoE-Lite implementation for MLX.
kev
Package kev implements the Qwen3.5/Qwen3.8 Kev decision head.
Package kev implements the Qwen3.5/Qwen3.8 Kev decision head.
laguna
Package laguna provides the Poolside Laguna text model implementation for MLX.
Package laguna provides the Poolside Laguna text model implementation for MLX.
laya
Package laya implements the ModernBERT decision model from https://github.com/NandhaKishorM/laya (Apache-2.0).
Package laya implements the ModernBERT decision model from https://github.com/NandhaKishorM/laya (Apache-2.0).
lfm3
Package lfm3 implements Liquid's LFM3 hybrid text model for MLX.
Package lfm3 implements Liquid's LFM3 hybrid text model for MLX.
lfm3/lfm3audio
Package lfm3audio implements the speech-input path shared by LFM3-Audio and LFM3-Omni: log-mel features, the strided-convolution downsampler, the LFM2 speech encoder, and the projector into the language model.
Package lfm3audio implements the speech-input path shared by LFM3-Audio and LFM3-Omni: log-mel features, the strided-convolution downsampler, the LFM2 speech encoder, and the projector into the language model.
lfm3/lfm3speech
Package lfm3speech implements the speech-output half of LFM3-Audio and LFM3-Omni: the audio-code feedback embedding, the MaskGIT depth decoder that turns a backbone hidden state into one frame of codec codes, and the LiquidCodec decoder that turns code frames into 24 kHz audio.
Package lfm3speech implements the speech-output half of LFM3-Audio and LFM3-Omni: the audio-code feedback embedding, the MaskGIT depth decoder that turns a backbone hidden state into one frame of codec codes, and the LiquidCodec decoder that turns code frames into 24 kHz audio.
llama
Package llama provides a Llama-style decoder-only transformer for MLX.
Package llama provides a Llama-style decoder-only transformer for MLX.
nemotron_h
Package nemotron_h provides the Nemotron-H text model implementation for MLX.
Package nemotron_h provides the Nemotron-H text model implementation for MLX.
pplxdecider
Package pplxdecider implements Perplexity's decision checkpoint on its Qwen3_5Model backbone.
Package pplxdecider implements Perplexity's decision checkpoint on its Qwen3_5Model backbone.
qwen3
Package qwen3 provides the Qwen3 text model implementation for MLX.
Package qwen3 provides the Qwen3 text model implementation for MLX.
qwen3_5
Package qwen3_5 provides the Qwen 3.5 text and MoE implementation for MLX.
Package qwen3_5 provides the Qwen 3.5 text and MoE implementation for MLX.
qwen3_5_moe
Package qwen3_5_moe registers Qwen 3.5 MoE architecture aliases.
Package qwen3_5_moe registers Qwen 3.5 MoE architecture aliases.
qwen4_exp
Package qwen4_exp implements the Qwen 4 MLX model family.
Package qwen4_exp implements the Qwen 4 MLX model family.
strands
Package strands implements Strands Decider's Qwen3.5 pointer-head model.
Package strands implements Strands Decider's Qwen3.5 pointer-head model.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL