create

package
v0.40.12 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 27, 2026 License: MIT Imports: 33 Imported by: 0

Documentation

Index

Constants

View Source
const SafetensorsMinOllamaVersion = "0.19.0"

SafetensorsMinOllamaVersion is the minimum Ollama version required for safetensors-backed models.

Variables

View Source
var (
	ErrBadTemplate     = errors.New("template error")
	ErrInvalidRequires = errors.New("invalid requires version")
	ErrInvalidLicense  = errors.New("license must be a string or a list of strings")
)
View Source
var ErrUnsupportedMLXArchitecture = errors.New("unsupported MLX architecture")

Functions

func ApplyModelfileLayers

func ApplyModelfileLayers(layers []manifest.Layer, opts ModelfileLayerOptions) ([]manifest.Layer, error)

ApplyModelfileLayers overlays explicit Modelfile values onto inherited layers. Singleton values are replaced, parameters are merged, and licenses are appended.

func Create

func Create(ctx context.Context, modelName, modelDir string, opts PipelineOptions, store BlobStore, writeManifest ManifestWriter, fn func(status string)) error

Create imports a safetensors model through the full pipeline: read the source into an inventory, classify it, plan the output blobs, write them through store, import the config files, and write the manifest. It is the shared local and server entry point; the caller supplies blob storage (store) and manifest assembly (writeManifest). store, writeManifest, and fn must be non-nil.

func ExpertGroupPrefix

func ExpertGroupPrefix(tensorName string) string

ExpertGroupPrefix returns the group prefix for expert tensors that should be packed together. For example:

  • "model.layers.1.mlp.experts.0.down_proj.weight" -> "model.layers.1.mlp.experts"
  • "model.layers.1.mlp.shared_experts.down_proj.weight" -> "model.layers.1.mlp.shared_experts"
  • "language_model.model.layers.1.mlp.switch_mlp.down_proj.weight" -> "language_model.model.layers.1.mlp.switch_mlp"
  • "model.layers.0.mlp.down_proj.weight" -> "" (dense layer, no experts)
  • "model.layers.1.mlp.gate.weight" -> "" (routing gate, not an expert)

func GetTensorQuantization

func GetTensorQuantization(name string, shape []int32, quantize string) string

GetTensorQuantization returns the appropriate quantization type for a tensor. Returns "" if the tensor should not be quantized.

func IsSafetensorsLLMModel

func IsSafetensorsLLMModel(modelName string) bool

IsSafetensorsLLMModel checks for a safetensors completion or decision model.

func IsSafetensorsModelDir

func IsSafetensorsModelDir(dir string) bool

IsSafetensorsModelDir checks for a supported model config and safetensors weights.

func LicenseStrings

func LicenseStrings(license any) ([]string, error)

LicenseStrings normalizes a create request license value, which may arrive as a string, []string, or JSON-decoded []any, into a list of license texts.

func ReaderWithContext

func ReaderWithContext(ctx context.Context, r io.Reader) io.Reader

ReaderWithContext stops reads after ctx is canceled.

func SafetensorsConfigFiles added in v0.40.0

func SafetensorsConfigFiles(dir string) ([]string, error)

SafetensorsConfigFiles lists source metadata for remote transfer and local JSON import. Auxiliary checkpoint subfolders are excluded.

func SafetensorsWeightFiles

func SafetensorsWeightFiles(dir string) ([]string, error)

SafetensorsWeightFiles returns the source shards and supported sidecars for transfer. Importing a sidecar still requires its model-specific importer.

Types

type BlobSpec

type BlobSpec struct {
	Name     string
	Tensors  []TensorSpec
	Metadata map[string]string
}

BlobSpec describes one output blob: its layer name, the tensors it contains, and its safetensors metadata. The planner builds these purely from the inventory and classification; the writer executes them and makes no decisions of its own.

func Plan

func Plan(inv Inventory, class Classification, policy quantizePolicy) ([]BlobSpec, error)

Plan turns an inventory and its classification into the ordered list of blobs to write. It reads no weight data and makes every decision here, so the writer that follows has nothing left to decide. The policy decides which weights are quantized and to what; pass defaultQuantPolicy{} for the generic policy.

type BlobStore

type BlobStore interface {
	WriteBlob(r io.Reader, mediaType, name string) (LayerInfo, error)
}

BlobStore stores a finished blob and returns its layer info. The writer produces the blob bytes; whether the target is local or remote is the store's concern.

type Classification

type Classification struct {
	Kind     SourceKind
	Quantize string
}

Classification is the decision about a source model: its kind and the effective quantization of the imported weights. Quantize may describe a requested conversion or a quantization already present in the source.

func Classify

func Classify(inv Inventory, requested string) (Classification, error)

Classify decides a source model's kind and resolves the effective quantization from the user's requested type, rejecting requests that are not allowed for the kind.

type Inventory

type Inventory struct {
	Dir       string
	Config    sourceModelConfig
	RawConfig json.RawMessage
	Tensors   map[string]SourceTensor
}

Inventory is the immutable result of reading a source model: every tensor indexed by name, plus the parsed config and the model directory. Reading source headers happens only here; the classify, plan, and write steps work entirely from this listing and never re-open a source header to make a decision. RawConfig holds the config.json bytes so architecture-specific factories can parse their own fields without re-opening the file.

func ReadInventory

func ReadInventory(dir string) (Inventory, error)

ReadInventory reads a source model directory into an Inventory: the config, the shard index, and every tensor's header. It reads no weight data. If the shard index references a tensor that cannot be found (a missing or truncated shard, e.g. a partial download), it fails rather than silently producing an incomplete model.

func (Inventory) Has

func (inv Inventory) Has(name string) bool

Has reports whether a tensor with the given name exists in the source.

type LayerInfo

type LayerInfo struct {
	Digest    string
	Size      int64
	MediaType string
	Name      string // Path-style name: "component/tensor" or "path/to/config.json"
}

LayerInfo holds metadata for a created layer.

func CreateDraftLayers

func CreateDraftLayers(ctx context.Context, modelDir, tensorPrefix, configPrefix, quantize string, validation MLXValidationOptions, store BlobStore, fn func(status string)) ([]LayerInfo, error)

CreateDraftLayers imports a draft (speculative-decoding / MTP assistant) safetensors model into prefixed tensor and config blobs and returns the layers WITHOUT writing a manifest — the caller folds them into the target model's manifest. A draft never stands alone; it always accompanies a target model named on the Modelfile's FROM line.

It runs the same read → classify → plan → write pipeline as Create. Output tensor names keep their source form, namespaced by tensorPrefix (e.g. "draft.") so they cannot collide with the target's tensors; config blobs are named under configPrefix (e.g. "draft/"). store and fn must be non-nil.

func WriteBlobs

func WriteBlobs(ctx context.Context, specs []BlobSpec, modelDir string, store BlobStore) ([]LayerInfo, error)

WriteBlobs executes a plan's blobs: for each blob it resolves the tensors' sources, produces the blob bytes, and stores the result.

type MLXValidationOptions

type MLXValidationOptions struct {
	Force   bool
	Warning func(string)
}

MLXValidationOptions controls failures that can be downgraded while developing support for a new model architecture.

type ManifestBlobStore

type ManifestBlobStore struct{}

ManifestBlobStore writes blobs to the local content-addressed manifest store.

func (ManifestBlobStore) WriteBlob

func (ManifestBlobStore) WriteBlob(r io.Reader, mediaType, name string) (LayerInfo, error)

type ManifestInfo

type ManifestInfo struct {
	ModelConfig model.ConfigV2
	ConfigLayer LayerInfo
	Layers      []LayerInfo
	Class       Classification
}

ManifestInfo is the data a manifest writer needs after the import pipeline has planned and written the source model.

type ManifestWriter

type ManifestWriter func(ctx context.Context, modelName string, info ManifestInfo) error

ManifestWriter writes the manifest file.

func NewSafetensorsManifestWriter

func NewSafetensorsManifestWriter(opts SafetensorsManifestOptions) ManifestWriter

NewSafetensorsManifestWriter returns a ManifestWriter that builds the shared safetensors config and Modelfile-derived manifest layers.

type ModelfileLayerOptions

type ModelfileLayerOptions struct {
	Template   string
	System     string
	License    any
	Parameters map[string]any
	Messages   []api.Message
}

ModelfileLayerOptions contains the Modelfile values that overlay inherited manifest layers during create.

type PipelineOptions

type PipelineOptions struct {
	Quantize      string
	Parser        string
	Renderer      string
	Requires      string
	DraftDir      string
	DraftQuantize string
	Validation    MLXValidationOptions
}

PipelineOptions controls the source-specific stages of a safetensors import.

type SafetensorsManifestOptions

type SafetensorsManifestOptions struct {
	MinVersion string
	DraftDir   string

	Template     string
	System       string
	License      any
	Parameters   map[string]any
	Messages     []api.Message
	Capabilities []string

	BeforeWriteManifest func()
}

SafetensorsManifestOptions describes the config and Modelfile-derived layers shared by local and server-side safetensors create.

type SourceKind

type SourceKind int

SourceKind is the overarching dtype for a given safetensors model

const (
	SourceFloat        SourceKind = iota // bf16/fp16/fp32 — quantizable on request
	SourceBlockFP8                       // HF block-FP8 — auto-converted to mxfp8
	SourcePrequantized                   // already quantized — copied through
)

func (SourceKind) String

func (k SourceKind) String() string

type SourceTensor

type SourceTensor struct {
	Name  string
	Dtype string
	Shape []int32
	File  string // safetensors file basename, relative to the model directory
}

SourceTensor describes one tensor found in a source model: its on-disk type and shape and which safetensors file holds it. It carries no weight data — only what the header and shard index reveal.

type TensorSpec

type TensorSpec struct {
	Name      string
	Sources   []SourceTensor
	Transform Transform
	Quantize  string
	OutDtype  string  // dtype after the transform; "" means same as the single source
	OutShape  []int32 // shape after the transform; nil means same as the single source
}

TensorSpec describes one output tensor within a blob: the source tensor(s) it is built from, the transform that combines or converts them, the name it takes in the blob, and an optional quantization to apply. When Quantize is set the writer runs MLX quantization, which generates the tensor's scale and bias sub-tensors; otherwise the (transformed) bytes are stored as-is.

type Transform

type Transform string

Transform names how a tensor's source(s) are turned into the output tensor. The zero value, TransformNone, copies a single source through unchanged.

const (
	TransformNone Transform = ""

	// TransformRepackFP4 reinterprets a U8 fp4-packed weight (2 values/byte)
	// as U32 words (8 values/word): the bytes are unchanged, only the dtype
	// and last dimension are relabeled.
	TransformRepackFP4 Transform = "repack_fp4"

	// TransformRepackFP8 reinterprets four FP8 values as one U32 word for
	// MLX's mxfp8 matrix multiplication. The bytes are unchanged.
	TransformRepackFP8 Transform = "repack_fp8"

	// TransformRelabelU8 relabels an F8_E4M3 scale as U8 so the loader reads
	// its raw bytes; the bytes themselves are unchanged.
	TransformRelabelU8 Transform = "relabel_u8"

	// TransformScalarF32 validates that the source is a scalar F32 and copies
	// it through (a global scale stored as-is).
	TransformScalarF32 Transform = "scalar_f32"

	// TransformF32 preserves an F32 vector scale bank with one value per expert.
	TransformF32 Transform = "f32"

	// TransformReciprocalF32 validates an F32 scale tensor and stores each
	// element's reciprocal (global scales the producer stored inverted).
	TransformReciprocalF32 Transform = "reciprocal_f32"

	// TransformStackExperts concatenates N per-expert source tensors (in
	// expert-index order) into one [experts, ...] tensor.
	TransformStackExperts Transform = "stack_experts"

	// TransformDecodeFP8 dequantizes a block-FP8 weight using its block scale.
	// Its two sources are the F8_E4M3 weight and its scale companion; the
	// result is a BF16 tensor, which Quantize (if set) then re-quantizes.
	TransformDecodeFP8 Transform = "decode_fp8"

	// TransformDecodeStackFP8 stacks N per-expert block-FP8 weights (and their N
	// block scales) into one [experts, out, in] tensor and dequantizes it. Its
	// sources are the N weights followed by the N scales, in expert-index order;
	// the result is a BF16 tensor, which Quantize (if set) then re-quantizes.
	TransformDecodeStackFP8 Transform = "decode_stack_fp8"
)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL