Documentation
¶
Overview ¶
Package gguf provides a reader for GGUF (GGML Universal Format) model files, with support for loading tensor data as GoMLX tensors, including on-the-fly dequantization of quantized weight formats.
Example loading from a local file:
model, err := gguf.NewFromFile("/path/to/model.gguf")
if err != nil {
panic(err)
}
for tn, err := range model.IterTensors() {
if err != nil {
panic(err)
}
fmt.Printf("- Tensor %s: shape=%s\n", tn.Name, tn.Tensor.Shape())
}
Example loading from HuggingFace:
repo := hub.New(modelID).WithAuth(hfAuthToken)
for tn, err := range gguf.IterTensorsFromRepo(repo) {
if err != nil {
panic(err)
}
fmt.Printf("- Tensor %s: shape=%s\n", tn.Name, tn.Tensor.Shape())
}
Index ¶
- Constants
- func IterTensorsFromRepo(backend compute.Backend, repo *hub.Repo) func(yield func(TensorAndName, error) bool)
- type File
- type KeyValue
- type Model
- func (m *Model) Architecture() string
- func (m *Model) Close() error
- func (m *Model) GetKeyValue(key string) (KeyValue, bool)
- func (m *Model) GetTensor(backend compute.Backend, tensorName string) (*TensorAndName, error)
- func (m *Model) IterTensors(backend compute.Backend) func(yield func(TensorAndName, error) bool)
- func (m *Model) ListTensorNames() []string
- func (m *Model) Load() error
- type Reader
- type TensorAndName
- type TensorInfo
- type TensorType
- type Value
- func (v Value) Bool() bool
- func (v Value) Float64() float64
- func (v Value) Float64s() []float64
- func (v Value) Int64() int64
- func (v Value) Int64s() []int64
- func (v Value) Raw() any
- func (v Value) String() string
- func (v Value) Strings() []string
- func (v Value) Uint64() uint64
- func (v Value) Uint64s() []uint64
Constants ¶
const ( // Well-known GGUF metadata keys from the specification. KeyGeneralArchitecture = "general.architecture" KeyGeneralAlignment = "general.alignment" )
Variables ¶
This section is empty.
Functions ¶
func IterTensorsFromRepo ¶
func IterTensorsFromRepo(backend compute.Backend, repo *hub.Repo) func(yield func(TensorAndName, error) bool)
IterTensorsFromRepo creates a Model from a repo and iterates over all tensors.
Tensors are loaded into the backend directly (e.g.: GPU, or a shared memory tensor on CPU, etc). If the backend is nil, it instead loads them in host memory.
Types ¶
type File ¶
type File struct {
// Version is the GGUF format version (2 or 3).
Version uint32
// Alignment is the byte alignment for tensor data (default 32).
Alignment uint64
// KeyValues holds all metadata key-value pairs from the file header.
KeyValues []KeyValue
// TensorInfos holds parsed information about every tensor in the file.
TensorInfos []TensorInfo
// contains filtered or unexported fields
}
File represents a parsed GGUF file. Create one with Open.
func Open ¶
Open opens and parses a GGUF file, reading all metadata and tensor info. The returned File can be used to look up metadata and read tensor data.
func (*File) Architecture ¶
Architecture returns the model architecture string (e.g., "llama", "gemma"), or "" if the metadata key "general.architecture" is not present.
func (*File) DataOffset ¶
DataOffset returns the byte offset where tensor data begins in the file.
func (*File) GetKeyValue ¶
GetKeyValue looks up a metadata key-value pair by its key.
func (*File) GetTensorInfo ¶
func (f *File) GetTensorInfo(name string) (TensorInfo, bool)
GetTensorInfo looks up a tensor by name.
func (*File) ListTensorNames ¶
ListTensorNames returns the names of all tensors in the file.
type Model ¶
Model represents a GGUF model, optionally backed by a HuggingFace repo.
func NewEmpty ¶
NewEmpty creates an empty Model for manual control. Call Load() to download and parse.
func NewFromFile ¶
NewFromFile creates a Model directly from a local GGUF file path.
func (*Model) Architecture ¶
Architecture returns the model architecture string.
func (*Model) GetKeyValue ¶
GetKeyValue looks up a metadata key-value pair.
func (*Model) IterTensors ¶
IterTensors returns an iterator over all tensors as GoMLX tensors. Tensors are read sequentially sorted by offset for optimal I/O.
Tensors are loaded into the backend directly (e.g.: GPU, or a shared memory tensor on CPU, etc). If the backend is nil, it instead loads them in host memory.
func (*Model) ListTensorNames ¶
ListTensorNames returns all tensor names in the model.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader provides random-access to tensor data in a GGUF file.
func (*Reader) ReadTensor ¶
ReadTensor reads a tensor by name: native types (F32, F16, BF16, I8, etc.) are loaded directly; GGUF quantized types are dequantized to Float32.
func (*Reader) ReadTensorRaw ¶
func (r *Reader) ReadTensorRaw(tensorName string) ([]byte, *TensorInfo, error)
ReadTensorRaw reads the raw bytes for a tensor without dequantization.
type TensorAndName ¶
TensorAndName holds a tensor name and its GoMLX tensor data.
type TensorInfo ¶
type TensorInfo struct {
Name string
Shape []uint64 // Dimensions in GGUF native order (innermost first).
Type TensorType
Offset uint64 // Byte offset within the tensor data section.
}
TensorInfo holds parsed information about a single tensor in a GGUF file.
func (*TensorInfo) GoMLXShape ¶
func (ti *TensorInfo) GoMLXShape() (dtypes.DType, []int)
GoMLXShape returns the GoMLX dtype and dimensions for this tensor. GGUF stores dimensions innermost-first; this reverses them to the outermost-first convention used by GoMLX and HuggingFace.
Quantized dtypes get converted to dtypes.Float32.
func (*TensorInfo) NumBytes ¶
func (ti *TensorInfo) NumBytes() int64
NumBytes returns the total number of bytes the tensor data occupies in the file. Returns 0 if the result would overflow int64.
func (*TensorInfo) NumElements ¶
func (ti *TensorInfo) NumElements() uint64
NumElements returns the total number of elements in the tensor.
type TensorType ¶
type TensorType uint32
TensorType represents the data type or quantization format of a tensor in a GGUF file.
const ( TensorTypeF32 TensorType = 0 TensorTypeF16 TensorType = 1 TensorTypeQ4_0 TensorType = 2 TensorTypeQ4_1 TensorType = 3 // 4, 5 are unused/removed types. TensorTypeQ5_0 TensorType = 6 TensorTypeQ5_1 TensorType = 7 TensorTypeQ8_0 TensorType = 8 TensorTypeQ8_1 TensorType = 9 TensorTypeQ2_K TensorType = 10 TensorTypeQ3_K TensorType = 11 TensorTypeQ4_K TensorType = 12 TensorTypeQ5_K TensorType = 13 TensorTypeQ6_K TensorType = 14 TensorTypeQ8_K TensorType = 15 TensorTypeIQ2_XXS TensorType = 16 TensorTypeIQ2_XS TensorType = 17 TensorTypeIQ3_XXS TensorType = 18 TensorTypeIQ1_S TensorType = 19 TensorTypeIQ4_NL TensorType = 20 TensorTypeIQ3_S TensorType = 21 TensorTypeIQ2_S TensorType = 22 TensorTypeIQ4_XS TensorType = 23 TensorTypeI8 TensorType = 24 TensorTypeI16 TensorType = 25 TensorTypeI32 TensorType = 26 TensorTypeI64 TensorType = 27 TensorTypeF64 TensorType = 28 TensorTypeIQ1_M TensorType = 29 TensorTypeBF16 TensorType = 30 // 31-33 are unused. TensorTypeTQ1_0 TensorType = 34 TensorTypeTQ2_0 TensorType = 35 // 36-38 are unused. TensorTypeMXFP4 TensorType = 39 )
func (TensorType) BlockSize ¶
func (t TensorType) BlockSize() int
BlockSize returns the number of elements per quantization block. Native types have a block size of 1. Legacy quantized types (Q4_0, Q8_0, etc.) have a block size of 32. K-quant types (Q2_K, Q4_K, etc.) have a block size of 256.
func (TensorType) GoMLXDType ¶
func (t TensorType) GoMLXDType() dtypes.DType
GoMLXDType returns the GoMLX dtype for native (non-quantized) tensor types. For quantized types, returns dtypes.Float32 (the dequantization output type).
func (TensorType) IsQuantized ¶
func (t TensorType) IsQuantized() bool
IsQuantized returns true if the tensor type requires dequantization to be used as standard floating-point data.
func (TensorType) String ¶
func (t TensorType) String() string
String returns a human-readable name for the tensor type.
func (TensorType) TypeSize ¶
func (t TensorType) TypeSize() int
TypeSize returns the number of bytes per quantization block. For native types with block size 1, this is the element size in bytes.
type Value ¶
type Value struct {
// contains filtered or unexported fields
}
Value wraps a GGUF metadata value with typed accessors. Accessors return zero values when the underlying type doesn't match, rather than returning errors.
func (Value) Float64 ¶
Float64 returns the value as a float64. Works for float32 and float64. Returns 0 if the value is not a float.
func (Value) Int64 ¶
Int64 returns the value as an int64. Works for any signed or unsigned integer type. Returns 0 if the value is not an integer.
func (Value) Int64s ¶
Int64s returns the value as an int64 slice, or nil if it is not an integer array.
Source Files