model

package
v0.35.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 28, 2026 License: MIT Imports: 22 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func GatherQMMIdentityScale

func GatherQMMIdentityScale() *mlx.Array

GatherQMMIdentityScale is the scale that leaves an expert bank unscaled, for rows folded into a scaled bank without a scale of their own.

func InferAffineQuantParamsFromShapes

func InferAffineQuantParamsFromShapes(weight, scales *mlx.Array, hintBits int) (groupSize, bits int, ok bool)

InferAffineQuantParamsFromShapes infers (groupSize,bits) for affine quantized tensors from packed weight and scale shapes.

func MakeEmbeddingLayer

func MakeEmbeddingLayer(
	tensors map[string]*mlx.Array,
	path string,
	defaultGroupSize, defaultBits int,
	defaultMode string,
	tensorQuant map[string]*TensorQuantInfo,
) nn.EmbeddingLayer

MakeEmbeddingLayer constructs an embedding layer from a tensor map.

For quantized tensors (path.weight + path.weight_scale), it returns a QuantizedEmbedding using the same quant metadata path that linear layers use. For non-quantized tensors, it returns a standard dense embedding.

func MakeLinearLayer

func MakeLinearLayer(
	tensors map[string]*mlx.Array,
	path string,
	defaultGroupSize, defaultBits int,
	defaultMode string,
	tensorQuant map[string]*TensorQuantInfo,
) nn.LinearLayer

MakeLinearLayer constructs a linear layer from a tensor map.

For quantized tensors (path.weight + path.weight_scale), it resolves per-tensor quant params via TensorQuant metadata (with shape-based affine fallback). For non-quantized tensors, it returns a standard nn.Linear.

func PrepareGatherQMMGlobalScale

func PrepareGatherQMMGlobalScale(globalScale *mlx.Array, numExperts int) *mlx.Array

PrepareGatherQMMGlobalScale broadcasts an already-converted global scale into the one-entry-per-expert bank gather_qmm wants. Materialized dense: the kernel indexes it by raw offset, and a broadcast view is one element of storage.

func QuantizationParams

func QuantizationParams(quantization string) (groupSize, bits int, mode string)

QuantizationParams returns default groupSize, bits, and mode for a quantization type. The values live in the shared mlx/quant package so the importer, the runtime loader, and `ollama show` agree on them.

func ReadGlobalScale

func ReadGlobalScale(tensors map[string]*mlx.Array, weightKeys ...string) (*mlx.Array, []string)

ReadGlobalScale returns a weight's NVFP4 global scale in MLX's representation, and the companion keys the caller should release. Candidate keys are tried in order, so pass the resolved tensor key before any base.

func Register

func Register(arch string, fn func(root *Root) (Model, error))

Register registers a model constructor by architecture name. Called from init() in model packages. Panics on duplicate registration.

func RegisterDraft

func RegisterDraft(arch string, fn func(root *Root, target Model) (DraftModel, error))

RegisterDraft registers a draft model constructor by architecture name.

func ResolveLinearQuantParams

func ResolveLinearQuantParams(
	defaultGroupSize, defaultBits int,
	defaultMode string,
	tensorQuant map[string]*TensorQuantInfo,
	tensorName string,
	weight, scales *mlx.Array,
) (groupSize, bits int, mode string)

ResolveLinearQuantParams resolves quantization params for a quantized linear tensor, preferring per-tensor metadata and falling back to shape-based inference for affine packed tensors.

func SameGlobalScales

func SameGlobalScales(a, b *mlx.Array) bool

SameGlobalScales reports whether two prepared banks hold the same scale for every expert, which is what lets two projections share one fused bank.

func SupportsArchitecture

func SupportsArchitecture(arch string) bool

SupportsArchitecture reports whether a target model constructor is registered.

func SupportsDraftArchitecture

func SupportsDraftArchitecture(arch string) bool

SupportsDraftArchitecture reports whether a draft model constructor is registered.

func TensorQuantParams

func TensorQuantParams(
	defaultGroupSize, defaultBits int,
	defaultMode string,
	tensorQuant map[string]*TensorQuantInfo,
	tensorName string,
) (groupSize, bits int, mode string, fromTensor bool)

TensorQuantParams resolves quant params for a tensor using per-tensor metadata when available, otherwise falling back to the provided model defaults.

func ToMLXGlobalScale

func ToMLXGlobalScale(globalScale *mlx.Array) *mlx.Array

ToMLXGlobalScale converts a checkpoint multiplier into the representation every global scale is held in once loaded. Shape is flattened too: a scalar ships as either [] or [1], and stacking a mix of the two fails.

Types

type BlockDraft

type BlockDraft interface {
	DraftModel

	// BlockParams returns the trained block length and the mask token
	// standing in for undrafted positions.
	BlockParams() (blockSize int, maskToken int32)
}

BlockDraft is a DraftModel that drafts a whole block per forward (block diffusion), conditioned on features tapped from target layers rather than the final hidden state.

type DraftModel

type DraftModel interface {
	// LoadWeights assigns manifest tensors to the draft model's fields. An
	// inline head has nothing to do here; its weights load with the target's.
	LoadWeights(tensors map[string]*mlx.Array) error

	// NewCaches builds the cache slots this draft model writes, or nil
	// when it keeps no KV.
	NewCaches() []cache.Cache

	// Forward consumes b.Hidden (the draft-conditioning state) and returns
	// its hidden plus the aux hidden that seeds the next step. targetCaches
	// is read-only, for drafts that attend over the target's history.
	Forward(b *batch.Batch, targetCaches, draftCaches []cache.Cache) (hidden, auxHidden *mlx.Array)

	// Unembed projects a hidden state to vocabulary logits.
	Unembed(x *mlx.Array) *mlx.Array
}

DraftModel is an auxiliary model alongside a target that proposes speculative tokens.

func NewDraft

func NewDraft(root *Root, target Model) (DraftModel, error)

NewDraft constructs the draft model described by the manifest config, if any.

type LinearFactory

type LinearFactory struct {
	// contains filtered or unexported fields
}

LinearFactory builds linear layers using shared tensor maps and quant defaults.

func NewLinearFactory

func NewLinearFactory(
	tensors map[string]*mlx.Array,
	defaultGroupSize, defaultBits int,
	defaultMode string,
	tensorQuant map[string]*TensorQuantInfo,
) LinearFactory

NewLinearFactory creates a reusable constructor for model linear layers.

func (LinearFactory) Make

func (f LinearFactory) Make(path string) nn.LinearLayer

Make constructs a linear layer at path.

type MediaModel

type MediaModel interface {
	// PrepareMedia runs once per request on the request goroutine, CPU
	// only, and returns the expanded stream. It must be deterministic for
	// given segments: prefix-cache restores splice cached state with
	// recomputed state.
	PrepareMedia(segments []Segment) (*PreparedRequest, error)

	// EncodeMedia builds one item's lazy feature graph on the MLX thread;
	// it must not evaluate — the consuming forward's evaluation pulls it.
	// Read the pixels from data: the runner frees the item's MediaData
	// once its expansion is evaluated.
	EncodeMedia(item *PreparedItem, data *mlx.Array) *mlx.Array
}

MediaModel is implemented by models that accept media inputs.

type Model

type Model interface {
	// LoadWeights receives all tensors loaded from the manifest and assigns
	// them to model fields. Model-specific logic (MLA absorption, expert
	// stacking, quantized layer creation) happens here.
	LoadWeights(tensors map[string]*mlx.Array) error

	// NewCaches builds the cache slots this model's layers need.
	NewCaches() []cache.Cache

	// Forward returns the hidden state to unembed and the state a draft
	// model conditions on; plain models return the final hidden for both.
	Forward(b *batch.Batch, cache []cache.Cache) (hidden, auxHidden *mlx.Array)
	Unembed(x *mlx.Array) *mlx.Array

	Tokenizer() *tokenizer.Tokenizer
	MaxContextLength() int
}

Model is the interface that model implementations must satisfy.

func New

func New(root *Root) (Model, error)

New reads config.json from the manifest, detects the architecture, looks up the registered constructor, and calls it to create the model (with config parsed and struct created, but weights not yet loaded).

type PreparedItem

type PreparedItem struct {
	// Range is the expansion's token range [start, end) in Tokens;
	// non-empty, since cache identity enters through these positions.
	Range [2]int

	// Source is the index of the segment this item was prepared from; the
	// item's prefix-cache identity is keyed on that segment's bytes.
	Source int

	// MediaData is the preprocessed encoder input with shape Dims. Dims
	// enters the cache keys too: geometry changes features under
	// identical bytes.
	MediaData []float32
	Dims      []int

	// Opaque carries model-private preprocessing state to EncodeMedia
	// and Forward.
	Opaque any

	// Causal marks an expansion whose tokens attend causally, so chunks
	// may split it. Unset, the first evaluation covers the whole
	// expansion in one forward, as bidirectional runs require.
	Causal bool
}

PreparedItem describes one media occurrence in the prepared stream. A model chooses item granularity: one per media segment, or several when parts encode and evaluate independently (e.g. per tile).

type PreparedRequest

type PreparedRequest struct {
	Tokens []int32
	Items  []PreparedItem

	// Layout is an opaque request-scoped value computed in the one pass
	// that sees every splice position; immutable, carried unread by the
	// runner to every forward. Delivered only when Items is non-empty.
	// Nil when the model derives nothing from it.
	Layout any
}

PreparedRequest is the expanded input stream, every media segment's expansion spliced in place, with the items in stream order.

type Root

type Root struct {
	Manifest *manifest.Manifest
	Draft    *modeltypes.Draft
	// contains filtered or unexported fields
}

Root wraps a model's manifest with pre-scanned quantization metadata.

func Open

func Open(modelName string) (*Root, error)

Open loads a manifest for the given model name and scans tensor blobs for quantization metadata.

func (*Root) AllTensorQuant

func (r *Root) AllTensorQuant() map[string]*TensorQuantInfo

AllTensorQuant returns a copy of the per-tensor quantization metadata.

func (*Root) GroupSize

func (r *Root) GroupSize() int

GroupSize returns the quantization group size detected from the first tensor blob metadata.

func (*Root) QuantType

func (r *Root) QuantType() string

QuantType returns the quantization type detected from the first tensor blob metadata.

func (*Root) TensorQuant

func (r *Root) TensorQuant(name string) *TensorQuantInfo

TensorQuant returns per-tensor quantization metadata if available.

type Segment

type Segment struct {
	Tokens []int32
	Kind   string
	Data   []byte
}

Segment is one run of the prompt in stream order: either a tokenized text run (Tokens set) or a single media item (Kind and Data set).

type SelfDraft

type SelfDraft interface {
	SelfDraft() DraftModel
}

SelfDraft is implemented by models whose draft head ships inline with the target weights; it returns the head, or nil when the checkpoint shipped none.

type TensorQuantInfo

type TensorQuantInfo struct {
	QuantType string
	GroupSize int
}

TensorQuantInfo describes per-tensor quantization metadata.

Directories

Path Synopsis
Package cohere2_moe provides the Cohere2 MoE (Command A family, North) text model implementation for MLX.
Package cohere2_moe provides the Cohere2 MoE (Command A family, North) text model implementation for MLX.
Package dflash implements the DFlash block-diffusion draft model: qwen3-shaped layers drafting a whole block per forward, conditioned on tapped target hidden states as key/value context.
Package dflash implements the DFlash block-diffusion draft model: qwen3-shaped layers drafting a whole block per forward, conditioned on tapped target hidden states as key/value context.
Package gemma4 provides the Gemma 4 model implementation for MLX.
Package gemma4 provides the Gemma 4 model implementation for MLX.
Package glimmer provides the Meta Glimmer text model implementation for MLX.
Package glimmer provides the Meta Glimmer text model implementation for MLX.
Package glm4_moe_lite provides the GLM4-MoE-Lite implementation for MLX.
Package glm4_moe_lite provides the GLM4-MoE-Lite implementation for MLX.
Package laguna provides the Poolside Laguna text model implementation for MLX.
Package laguna provides the Poolside Laguna text model implementation for MLX.
Package llama provides a Llama-style decoder-only transformer for MLX.
Package llama provides a Llama-style decoder-only transformer for MLX.
Package nemotron_h provides the Nemotron-H text model implementation for MLX.
Package nemotron_h provides the Nemotron-H text model implementation for MLX.
Package qwen3 provides the Qwen3 text model implementation for MLX.
Package qwen3 provides the Qwen3 text model implementation for MLX.
Package qwen3_5 provides the Qwen 3.5 text and MoE implementation for MLX.
Package qwen3_5 provides the Qwen 3.5 text and MoE implementation for MLX.
Package qwen3_5_moe registers Qwen 3.5 MoE architecture aliases.
Package qwen3_5_moe registers Qwen 3.5 MoE architecture aliases.
Package qwen4_exp implements the Qwen 4 MLX model family.
Package qwen4_exp implements the Qwen 4 MLX model family.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL