Affected by 
Documentation
¶
Index ¶
- func InferAffineQuantParamsFromShapes(weight, scales *mlx.Array, hintBits int) (groupSize, bits int, ok bool)
- func MakeEmbeddingLayer(tensors map[string]*mlx.Array, path string, defaultGroupSize, defaultBits int, ...) nn.EmbeddingLayer
- func MakeLinearLayer(tensors map[string]*mlx.Array, path string, defaultGroupSize, defaultBits int, ...) nn.LinearLayer
- func QuantizationParams(quantization string) (groupSize, bits int, mode string)
- func ResolveLinearQuantParams(defaultGroupSize, defaultBits int, defaultMode string, ...) (groupSize, bits int, mode string)
- func TensorQuantParams(defaultGroupSize, defaultBits int, defaultMode string, ...) (groupSize, bits int, mode string, fromTensor bool)
- type LinearFactory
- type Root
- type TensorQuantInfo
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func InferAffineQuantParamsFromShapes ¶
func InferAffineQuantParamsFromShapes(weight, scales *mlx.Array, hintBits int) (groupSize, bits int, ok bool)
InferAffineQuantParamsFromShapes infers (groupSize,bits) for affine quantized tensors from packed weight and scale shapes.
func MakeEmbeddingLayer ¶ added in v0.18.2
func MakeEmbeddingLayer( tensors map[string]*mlx.Array, path string, defaultGroupSize, defaultBits int, defaultMode string, tensorQuant map[string]*TensorQuantInfo, ) nn.EmbeddingLayer
MakeEmbeddingLayer constructs an embedding layer from a tensor map.
For quantized tensors (path.weight + path.weight_scale), it returns a QuantizedEmbedding using the same quant metadata path that linear layers use. For non-quantized tensors, it returns a standard dense embedding.
func MakeLinearLayer ¶
func MakeLinearLayer( tensors map[string]*mlx.Array, path string, defaultGroupSize, defaultBits int, defaultMode string, tensorQuant map[string]*TensorQuantInfo, ) nn.LinearLayer
MakeLinearLayer constructs a linear layer from a tensor map.
For quantized tensors (path.weight + path.weight_scale), it resolves per-tensor quant params via TensorQuant metadata (with shape-based affine fallback). For non-quantized tensors, it returns a standard nn.Linear.
func QuantizationParams ¶
QuantizationParams returns default groupSize, bits, and mode for a quantization type. The values live in the shared x/quant package so the importer, the runtime loader, and `ollama show` agree on them.
func ResolveLinearQuantParams ¶
func ResolveLinearQuantParams( defaultGroupSize, defaultBits int, defaultMode string, tensorQuant map[string]*TensorQuantInfo, tensorName string, weight, scales *mlx.Array, ) (groupSize, bits int, mode string)
ResolveLinearQuantParams resolves quantization params for a quantized linear tensor, preferring per-tensor metadata and falling back to shape-based inference for affine packed tensors.
func TensorQuantParams ¶
func TensorQuantParams( defaultGroupSize, defaultBits int, defaultMode string, tensorQuant map[string]*TensorQuantInfo, tensorName string, ) (groupSize, bits int, mode string, fromTensor bool)
TensorQuantParams resolves quant params for a tensor using per-tensor metadata when available, otherwise falling back to the provided model defaults.
Types ¶
type LinearFactory ¶
type LinearFactory struct {
// contains filtered or unexported fields
}
LinearFactory builds linear layers using shared tensor maps and quant defaults.
func NewLinearFactory ¶
func NewLinearFactory( tensors map[string]*mlx.Array, defaultGroupSize, defaultBits int, defaultMode string, tensorQuant map[string]*TensorQuantInfo, ) LinearFactory
NewLinearFactory creates a reusable constructor for model linear layers.
func (LinearFactory) Make ¶
func (f LinearFactory) Make(path string) nn.LinearLayer
Make constructs a linear layer at path.
type Root ¶
type Root struct {
Manifest *manifest.ModelManifest
Draft *modeltypes.Draft
// contains filtered or unexported fields
}
Root wraps a ModelManifest with pre-scanned quantization metadata.
func Open ¶
Open loads a manifest for the given model name and scans tensor blobs for quantization metadata.
func (*Root) AllTensorQuant ¶
func (r *Root) AllTensorQuant() map[string]*TensorQuantInfo
AllTensorQuant returns a copy of the per-tensor quantization metadata.
func (*Root) GroupSize ¶
GroupSize returns the quantization group size detected from the first tensor blob metadata.
func (*Root) QuantType ¶
QuantType returns the quantization type detected from the first tensor blob metadata.
func (*Root) TensorQuant ¶
func (r *Root) TensorQuant(name string) *TensorQuantInfo
TensorQuant returns per-tensor quantization metadata if available.
type TensorQuantInfo ¶
TensorQuantInfo describes per-tensor quantization metadata.
Source Files
Directories