Documentation
¶
Index ¶
- func GatherQMMIdentityScale() *mlx.Array
- func InferAffineQuantParamsFromShapes(weight, scales *mlx.Array, hintBits int) (groupSize, bits int, ok bool)
- func MakeEmbeddingLayer(tensors map[string]*mlx.Array, path string, defaultGroupSize, defaultBits int, ...) nn.EmbeddingLayer
- func MakeLinearLayer(tensors map[string]*mlx.Array, path string, defaultGroupSize, defaultBits int, ...) nn.LinearLayer
- func PrepareGatherQMMGlobalScale(globalScale *mlx.Array, numExperts int) *mlx.Array
- func QuantizationParams(quantization string) (groupSize, bits int, mode string)
- func ReadGlobalScale(tensors map[string]*mlx.Array, weightKeys ...string) (*mlx.Array, []string)
- func Register(arch string, fn func(root *Root) (Model, error))
- func RegisterDraft(arch string, fn func(root *Root, target Model) (DraftModel, error))
- func ResolveLinearQuantParams(defaultGroupSize, defaultBits int, defaultMode string, ...) (groupSize, bits int, mode string)
- func SameGlobalScales(a, b *mlx.Array) bool
- func SupportsArchitecture(arch string) bool
- func SupportsDraftArchitecture(arch string) bool
- func TensorQuantParams(defaultGroupSize, defaultBits int, defaultMode string, ...) (groupSize, bits int, mode string, fromTensor bool)
- func ToMLXGlobalScale(globalScale *mlx.Array) *mlx.Array
- type BlockDraft
- type CachedScorer
- type DraftModel
- type LinearFactory
- type MediaModel
- type Model
- type PreparedItem
- type PreparedRequest
- type Root
- type ScoreForward
- type Segment
- type SelfDraft
- type TensorQuantInfo
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func GatherQMMIdentityScale ¶
GatherQMMIdentityScale is the scale that leaves an expert bank unscaled, for rows folded into a scaled bank without a scale of their own.
func InferAffineQuantParamsFromShapes ¶
func InferAffineQuantParamsFromShapes(weight, scales *mlx.Array, hintBits int) (groupSize, bits int, ok bool)
InferAffineQuantParamsFromShapes infers (groupSize,bits) for affine quantized tensors from packed weight and scale shapes.
func MakeEmbeddingLayer ¶
func MakeEmbeddingLayer( tensors map[string]*mlx.Array, path string, defaultGroupSize, defaultBits int, defaultMode string, tensorQuant map[string]*TensorQuantInfo, ) nn.EmbeddingLayer
MakeEmbeddingLayer constructs an embedding layer from a tensor map.
For quantized tensors (path.weight + path.weight_scale), it returns a QuantizedEmbedding using the same quant metadata path that linear layers use. For non-quantized tensors, it returns a standard dense embedding.
func MakeLinearLayer ¶
func MakeLinearLayer( tensors map[string]*mlx.Array, path string, defaultGroupSize, defaultBits int, defaultMode string, tensorQuant map[string]*TensorQuantInfo, ) nn.LinearLayer
MakeLinearLayer constructs a linear layer from a tensor map.
For quantized tensors (path.weight + path.weight_scale), it resolves per-tensor quant params via TensorQuant metadata (with shape-based affine fallback). For non-quantized tensors, it returns a standard nn.Linear.
func PrepareGatherQMMGlobalScale ¶
PrepareGatherQMMGlobalScale broadcasts an already-converted global scale into the one-entry-per-expert bank gather_qmm wants. Materialized dense: the kernel indexes it by raw offset, and a broadcast view is one element of storage.
func QuantizationParams ¶
QuantizationParams returns default groupSize, bits, and mode for a quantization type. The values live in the shared mlx/quant package so the importer, the runtime loader, and `ollama show` agree on them.
func ReadGlobalScale ¶
ReadGlobalScale returns a weight's NVFP4 global scale in MLX's representation, and the companion keys the caller should release. Candidate keys are tried in order, so pass the resolved tensor key before any base.
func Register ¶
Register registers a model constructor by architecture name. Called from init() in model packages. Panics on duplicate registration.
func RegisterDraft ¶
func RegisterDraft(arch string, fn func(root *Root, target Model) (DraftModel, error))
RegisterDraft registers a draft model constructor by architecture name.
func ResolveLinearQuantParams ¶
func ResolveLinearQuantParams( defaultGroupSize, defaultBits int, defaultMode string, tensorQuant map[string]*TensorQuantInfo, tensorName string, weight, scales *mlx.Array, ) (groupSize, bits int, mode string)
ResolveLinearQuantParams resolves quantization params for a quantized linear tensor, preferring per-tensor metadata and falling back to shape-based inference for affine packed tensors.
func SameGlobalScales ¶
SameGlobalScales reports whether two prepared banks hold the same scale for every expert, which is what lets two projections share one fused bank.
func SupportsArchitecture ¶
SupportsArchitecture reports whether a target model constructor is registered.
func SupportsDraftArchitecture ¶
SupportsDraftArchitecture reports whether a draft model constructor is registered.
func TensorQuantParams ¶
func TensorQuantParams( defaultGroupSize, defaultBits int, defaultMode string, tensorQuant map[string]*TensorQuantInfo, tensorName string, ) (groupSize, bits int, mode string, fromTensor bool)
TensorQuantParams resolves quant params for a tensor using per-tensor metadata when available, otherwise falling back to the provided model defaults.
Types ¶
type BlockDraft ¶
type BlockDraft interface {
DraftModel
// BlockParams returns the trained block length and the mask token
// standing in for undrafted positions.
BlockParams() (blockSize int, maskToken int32)
}
BlockDraft is a DraftModel that drafts a whole block per forward (block diffusion), conditioned on features tapped from target layers rather than the final hidden state.
type CachedScorer ¶ added in v0.40.0
type CachedScorer interface {
Score(context.Context, llm.ScoreRequest, ScoreForward) (llm.ScoreResponse, error)
}
CachedScorer owns decision preparation and readout; the runner owns forward execution and retained state. Bidirectional encoders cannot use this path.
type DraftModel ¶
type DraftModel interface {
// LoadWeights assigns manifest tensors to the draft model's fields. An
// inline head has nothing to do here; its weights load with the target's.
LoadWeights(tensors map[string]*mlx.Array) error
// NewCaches builds the cache slots this draft model writes, or nil
// when it keeps no KV.
NewCaches() []cache.Cache
// Forward consumes b.Hidden (the draft-conditioning state) and returns
// its hidden plus the aux hidden that seeds the next step. targetCaches
// is read-only, for drafts that attend over the target's history.
Forward(b *batch.Batch, targetCaches, draftCaches []cache.Cache) (hidden, auxHidden *mlx.Array)
// Unembed projects a hidden state to vocabulary logits.
Unembed(x *mlx.Array) *mlx.Array
}
DraftModel is an auxiliary model alongside a target that proposes speculative tokens.
type LinearFactory ¶
type LinearFactory struct {
// contains filtered or unexported fields
}
LinearFactory builds linear layers using shared tensor maps and quant defaults.
func NewLinearFactory ¶
func NewLinearFactory( tensors map[string]*mlx.Array, defaultGroupSize, defaultBits int, defaultMode string, tensorQuant map[string]*TensorQuantInfo, ) LinearFactory
NewLinearFactory creates a reusable constructor for model linear layers.
func (LinearFactory) Make ¶
func (f LinearFactory) Make(path string) nn.LinearLayer
Make constructs a linear layer at path.
type MediaModel ¶
type MediaModel interface {
// PrepareMedia runs once per request on the request goroutine, CPU
// only, and returns the expanded stream. It must be deterministic for
// given segments: prefix-cache restores splice cached state with
// recomputed state.
PrepareMedia(segments []Segment) (*PreparedRequest, error)
// EncodeMedia builds one item's lazy feature graph on the MLX thread;
// it must not evaluate — the consuming forward's evaluation pulls it.
// Read the pixels from data: the runner frees the item's MediaData
// once its expansion is evaluated.
EncodeMedia(item *PreparedItem, data *mlx.Array) *mlx.Array
}
MediaModel is implemented by models that accept media inputs.
type Model ¶
type Model interface {
// LoadWeights receives all tensors loaded from the manifest and assigns
// them to model fields. Model-specific logic (MLA absorption, expert
// stacking, quantized layer creation) happens here.
LoadWeights(tensors map[string]*mlx.Array) error
// NewCaches builds the cache slots this model's layers need.
NewCaches() []cache.Cache
// Forward returns the hidden state to unembed and the state a draft
// model conditions on; plain models return the final hidden for both.
Forward(b *batch.Batch, cache []cache.Cache) (hidden, auxHidden *mlx.Array)
Unembed(x *mlx.Array) *mlx.Array
Tokenizer() *tokenizer.Tokenizer
MaxContextLength() int
}
Model is the interface that model implementations must satisfy.
type PreparedItem ¶
type PreparedItem struct {
// Range is the expansion's token range [start, end) in Tokens;
// non-empty, since cache identity enters through these positions.
Range [2]int
// Source is the index of the segment this item was prepared from; the
// item's prefix-cache identity is keyed on that segment's bytes.
Source int
// MediaData is the preprocessed encoder input with shape Dims. Dims
// enters the cache keys too: geometry changes features under
// identical bytes.
MediaData []float32
Dims []int
// Opaque carries model-private preprocessing state to EncodeMedia
// and Forward.
Opaque any
// Causal marks an expansion whose tokens attend causally, so chunks
// may split it. Unset, the first evaluation covers the whole
// expansion in one forward, as bidirectional runs require.
Causal bool
}
PreparedItem describes one media occurrence in the prepared stream. A model chooses item granularity: one per media segment, or several when parts encode and evaluate independently (e.g. per tile).
type PreparedRequest ¶
type PreparedRequest struct {
Tokens []int32
Items []PreparedItem
// ScorePrefix is an optional token boundary shared by independent scoring
// rows. The scorer asks the runner to retain its cache state there.
ScorePrefix int
// Layout is an opaque request-scoped value computed in the one pass
// that sees every splice position; immutable, carried unread by the
// runner to every forward. Delivered only when Items is non-empty.
// Nil when the model derives nothing from it.
Layout any
}
PreparedRequest is the expanded input stream, every media segment's expansion spliced in place, with the items in stream order.
type Root ¶
type Root struct {
Manifest *manifest.Manifest
Draft *modeltypes.Draft
// contains filtered or unexported fields
}
Root wraps a model's manifest with pre-scanned quantization metadata.
func Open ¶
Open loads a manifest for the given model name and scans tensor blobs for quantization metadata.
func (*Root) AllTensorQuant ¶
func (r *Root) AllTensorQuant() map[string]*TensorQuantInfo
AllTensorQuant returns a copy of the per-tensor quantization metadata.
func (*Root) GroupSize ¶
GroupSize returns the quantization group size detected from the first tensor blob metadata.
func (*Root) QuantType ¶
QuantType returns the quantization type detected from the first tensor blob metadata.
func (*Root) TensorQuant ¶
func (r *Root) TensorQuant(name string) *TensorQuantInfo
TensorQuant returns per-tensor quantization metadata if available.
type ScoreForward ¶ added in v0.40.0
ScoreForward evaluates a causal backbone through the runner's prefix cache, returning the complete hidden row and the number of restored tokens. Segments supply the original media bytes for cache identity. Call within an MLX scope.
type Segment ¶
Segment is one run of the prompt in stream order: either a tokenized text run (Tokens set) or a single media item (Kind and Data set).
type SelfDraft ¶
type SelfDraft interface {
SelfDraft() DraftModel
}
SelfDraft is implemented by models whose draft head ships inline with the target weights; it returns the head, or nil when the checkpoint shipped none.
type TensorQuantInfo ¶
TensorQuantInfo describes per-tensor quantization metadata.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package clef implements Cloudflare's Qwen3.5 joint-schema decision model.
|
Package clef implements Cloudflare's Qwen3.5 joint-schema decision model. |
|
Package cohere2_moe provides the Cohere2 MoE (Command A family, North) text model implementation for MLX.
|
Package cohere2_moe provides the Cohere2 MoE (Command A family, North) text model implementation for MLX. |
|
Package dflash implements the DFlash block-diffusion draft model: qwen3-shaped layers drafting a whole block per forward, conditioned on tapped target hidden states as key/value context.
|
Package dflash implements the DFlash block-diffusion draft model: qwen3-shaped layers drafting a whole block per forward, conditioned on tapped target hidden states as key/value context. |
|
Package gemma4 provides the Gemma 4 model implementation for MLX.
|
Package gemma4 provides the Gemma 4 model implementation for MLX. |
|
Package gemma4embedding implements the Gemma4Embedding (EmbeddingGemma v2) text embedding model for MLX.
|
Package gemma4embedding implements the Gemma4Embedding (EmbeddingGemma v2) text embedding model for MLX. |
|
Package glimmer provides the Meta Glimmer text model implementation for MLX.
|
Package glimmer provides the Meta Glimmer text model implementation for MLX. |
|
Package glm4_moe_lite provides the GLM4-MoE-Lite implementation for MLX.
|
Package glm4_moe_lite provides the GLM4-MoE-Lite implementation for MLX. |
|
Package kev implements the Qwen3.5/Qwen3.8 Kev decision head.
|
Package kev implements the Qwen3.5/Qwen3.8 Kev decision head. |
|
Package laguna provides the Poolside Laguna text model implementation for MLX.
|
Package laguna provides the Poolside Laguna text model implementation for MLX. |
|
Package laya implements the ModernBERT decision model from https://github.com/NandhaKishorM/laya (Apache-2.0).
|
Package laya implements the ModernBERT decision model from https://github.com/NandhaKishorM/laya (Apache-2.0). |
|
Package llama provides a Llama-style decoder-only transformer for MLX.
|
Package llama provides a Llama-style decoder-only transformer for MLX. |
|
Package nemotron_h provides the Nemotron-H text model implementation for MLX.
|
Package nemotron_h provides the Nemotron-H text model implementation for MLX. |
|
Package pplxdecider implements Perplexity's decision checkpoint on its Qwen3_5Model backbone.
|
Package pplxdecider implements Perplexity's decision checkpoint on its Qwen3_5Model backbone. |
|
Package qwen3 provides the Qwen3 text model implementation for MLX.
|
Package qwen3 provides the Qwen3 text model implementation for MLX. |
|
Package qwen3_5 provides the Qwen 3.5 text and MoE implementation for MLX.
|
Package qwen3_5 provides the Qwen 3.5 text and MoE implementation for MLX. |
|
Package qwen3_5_moe registers Qwen 3.5 MoE architecture aliases.
|
Package qwen3_5_moe registers Qwen 3.5 MoE architecture aliases. |
|
Package qwen4_exp implements the Qwen 4 MLX model family.
|
Package qwen4_exp implements the Qwen 4 MLX model family. |
|
Package strands implements Strands Decider's Qwen3.5 pointer-head model.
|
Package strands implements Strands Decider's Qwen3.5 pointer-head model. |