create

package
v0.34.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 15, 2026 License: MIT Imports: 31 Imported by: 0

Documentation

Index

Constants

View Source 
const SafetensorsMinOllamaVersion = "0.19.0"

SafetensorsMinOllamaVersion is the minimum Ollama version required for safetensors-backed models.

Variables

View Source 
var (
	ErrBadTemplate     = errors.New("template error")
	ErrInvalidRequires = errors.New("invalid requires version")
	ErrInvalidLicense  = errors.New("license must be a string or a list of strings")
)
View Source 
var ErrUnsupportedMLXArchitecture = errors.New("unsupported MLX architecture")

Functions

func ApplyModelfileLayers added in v0.34.1 

func ApplyModelfileLayers(layers []manifest.Layer, opts ModelfileLayerOptions) ([]manifest.Layer, error)

ApplyModelfileLayers overlays explicit Modelfile values onto inherited layers. Singleton values are replaced, parameters are merged, and licenses are appended.

func Create added in v0.31.2 

func Create(ctx context.Context, modelName, modelDir string, opts PipelineOptions, store BlobStore, writeManifest ManifestWriter, fn func(status string)) error

Create imports a safetensors model through the full pipeline: read the source into an inventory, classify it, plan the output blobs, write them through store, import the config files, and write the manifest. It is the shared local and server entry point; the caller supplies blob storage (store) and manifest assembly (writeManifest). store, writeManifest, and fn must be non-nil.

func ExpertGroupPrefix added in v0.16.0 

func ExpertGroupPrefix(tensorName string) string

ExpertGroupPrefix returns the group prefix for expert tensors that should be packed together. For example:

  • "model.layers.1.mlp.experts.0.down_proj.weight" -> "model.layers.1.mlp.experts"
  • "model.layers.1.mlp.shared_experts.down_proj.weight" -> "model.layers.1.mlp.shared_experts"
  • "language_model.model.layers.1.mlp.switch_mlp.down_proj.weight" -> "language_model.model.layers.1.mlp.switch_mlp"
  • "model.layers.0.mlp.down_proj.weight" -> "" (dense layer, no experts)
  • "model.layers.1.mlp.gate.weight" -> "" (routing gate, not an expert)

func GetTensorQuantization added in v0.15.5 

func GetTensorQuantization(name string, shape []int32, quantize string) string

GetTensorQuantization returns the appropriate quantization type for a tensor. Returns "" if the tensor should not be quantized.

func IsSafetensorsLLMModel 

func IsSafetensorsLLMModel(modelName string) bool

IsSafetensorsLLMModel checks if a model is a safetensors LLM model (has completion capability, not image generation).

func IsSafetensorsModelDir 

func IsSafetensorsModelDir(dir string) bool

IsSafetensorsModelDir checks if the directory contains a standard safetensors model by looking for config.json and at least one .safetensors file.

func LicenseStrings added in v0.34.1 

func LicenseStrings(license any) ([]string, error)

LicenseStrings normalizes a create request license value, which may arrive as a string, []string, or JSON-decoded []any, into a list of license texts.

func ReaderWithContext added in v0.34.1 

func ReaderWithContext(ctx context.Context, r io.Reader) io.Reader

ReaderWithContext stops reads after ctx is canceled.

func SafetensorsWeightFiles added in v0.34.1 

func SafetensorsWeightFiles(dir string) ([]string, error)

SafetensorsWeightFiles returns the weight shards selected by the same index rules used by ReadInventory. Callers that transfer a source model should use this list rather than guessing shard names.

Types

type BlobSpec added in v0.31.2 

type BlobSpec struct {
	Name     string
	Tensors  []TensorSpec
	Metadata map[string]string
}

BlobSpec describes one output blob: its layer name, the tensors it contains, and its safetensors metadata. The planner builds these purely from the inventory and classification; the writer executes them and makes no decisions of its own.

func Plan added in v0.31.2 

func Plan(inv Inventory, class Classification, policy quantizePolicy) ([]BlobSpec, error)

Plan turns an inventory and its classification into the ordered list of blobs to write. It reads no weight data and makes every decision here, so the writer that follows has nothing left to decide. The policy decides which weights are quantized and to what; pass defaultQuantPolicy{} for the generic policy.

type BlobStore added in v0.31.2 

type BlobStore interface {
	WriteBlob(r io.Reader, mediaType, name string) (LayerInfo, error)
}

BlobStore stores a finished blob and returns its layer info. The writer produces the blob bytes; whether the target is local or remote is the store's concern.

type Classification added in v0.31.2 

type Classification struct {
	Kind     SourceKind
	Quantize string
}

Classification is the decision about a source model: its kind and the effective quantization of the imported weights. Quantize may describe a requested conversion or a quantization already present in the source.

func Classify added in v0.31.2 

func Classify(inv Inventory, requested string) (Classification, error)

Classify decides a source model's kind and resolves the effective quantization from the user's requested type, rejecting requests that are not allowed for the kind.

type Inventory added in v0.31.2 

type Inventory struct {
	Dir       string
	Config    sourceModelConfig
	RawConfig json.RawMessage
	Tensors   map[string]SourceTensor
}

Inventory is the immutable result of reading a source model: every tensor indexed by name, plus the parsed config and the model directory. Reading source headers happens only here; the classify, plan, and write steps work entirely from this listing and never re-open a source header to make a decision. RawConfig holds the config.json bytes so architecture-specific factories can parse their own fields without re-opening the file.

func ReadInventory added in v0.31.2 

func ReadInventory(dir string) (Inventory, error)

ReadInventory reads a source model directory into an Inventory: the config, the shard index, and every tensor's header. It reads no weight data. If the shard index references a tensor that cannot be found (a missing or truncated shard, e.g. a partial download), it fails rather than silently producing an incomplete model.

func (Inventory) Has added in v0.31.2 

func (inv Inventory) Has(name string) bool

Has reports whether a tensor with the given name exists in the source.

type LayerInfo 

type LayerInfo struct {
	Digest    string
	Size      int64
	MediaType string
	Name      string // Path-style name: "component/tensor" or "path/to/config.json"
}

LayerInfo holds metadata for a created layer.

func CreateDraftLayers added in v0.31.2 

func CreateDraftLayers(ctx context.Context, modelDir, tensorPrefix, configPrefix, quantize string, validation MLXValidationOptions, store BlobStore, fn func(status string)) ([]LayerInfo, error)

CreateDraftLayers imports a draft (speculative-decoding / MTP assistant) safetensors model into prefixed tensor and config blobs and returns the layers WITHOUT writing a manifest — the caller folds them into the target model's manifest. A draft never stands alone; it always accompanies a target model named on the Modelfile's FROM line.

It runs the same read → classify → plan → write pipeline as Create. Output tensor names keep their source form, namespaced by tensorPrefix (e.g. "draft.") so they cannot collide with the target's tensors; config blobs are named under configPrefix (e.g. "draft/"). store and fn must be non-nil.

func WriteBlobs added in v0.31.2 

func WriteBlobs(ctx context.Context, specs []BlobSpec, modelDir string, store BlobStore) ([]LayerInfo, error)

WriteBlobs executes a plan's blobs: for each blob it resolves the tensors' sources, produces the blob bytes, and stores the result.

type MLXValidationOptions added in v0.34.1 

type MLXValidationOptions struct {
	Force   bool
	Warning func(string)
}

MLXValidationOptions controls failures that can be downgraded while developing support for a new model architecture.

type ManifestBlobStore added in v0.34.1 

type ManifestBlobStore struct{}

ManifestBlobStore writes blobs to the local content-addressed manifest store.

func (ManifestBlobStore) WriteBlob added in v0.34.1 

func (ManifestBlobStore) WriteBlob(r io.Reader, mediaType, name string) (LayerInfo, error)

type ManifestInfo added in v0.34.1 

type ManifestInfo struct {
	ModelConfig model.ConfigV2
	ConfigLayer LayerInfo
	Layers      []LayerInfo
	Class       Classification
}

ManifestInfo is the data a manifest writer needs after the import pipeline has planned and written the source model.

type ManifestWriter 

type ManifestWriter func(ctx context.Context, modelName string, info ManifestInfo) error

ManifestWriter writes the manifest file.

func NewSafetensorsManifestWriter added in v0.34.1 

func NewSafetensorsManifestWriter(opts SafetensorsManifestOptions) ManifestWriter

NewSafetensorsManifestWriter returns a ManifestWriter that builds the shared safetensors config and Modelfile-derived manifest layers.

type ModelfileLayerOptions added in v0.34.1 

type ModelfileLayerOptions struct {
	Template   string
	System     string
	License    any
	Parameters map[string]any
	Messages   []api.Message
}

ModelfileLayerOptions contains the Modelfile values that overlay inherited manifest layers during create.

type PipelineOptions added in v0.34.1 

type PipelineOptions struct {
	Quantize      string
	Parser        string
	Renderer      string
	Requires      string
	DraftDir      string
	DraftQuantize string
	Validation    MLXValidationOptions
}

PipelineOptions controls the source-specific stages of a safetensors import.

type SafetensorsManifestOptions added in v0.34.1 

type SafetensorsManifestOptions struct {
	MinVersion string
	DraftDir   string

	Template   string
	System     string
	License    any
	Parameters map[string]any
	Messages   []api.Message

	BeforeWriteManifest func()
}

SafetensorsManifestOptions describes the config and Modelfile-derived layers shared by local and server-side safetensors create.

type SourceKind added in v0.31.2 

type SourceKind int

SourceKind is the overarching dtype for a given safetensors model 

const (
	SourceFloat        SourceKind = iota // bf16/fp16/fp32 — quantizable on request
	SourceBlockFP8                       // HF block-FP8 — auto-converted to mxfp8
	SourcePrequantized                   // already quantized — copied through
)

func (SourceKind) String added in v0.31.2 

func (k SourceKind) String() string

type SourceTensor added in v0.31.2 

type SourceTensor struct {
	Name  string
	Dtype string
	Shape []int32
	File  string // safetensors file basename, relative to the model directory
}

SourceTensor describes one tensor found in a source model: its on-disk type and shape and which safetensors file holds it. It carries no weight data — only what the header and shard index reveal.

type TensorSpec added in v0.31.2 

type TensorSpec struct {
	Name      string
	Sources   []SourceTensor
	Transform Transform
	Quantize  string
	OutDtype  string  // dtype after the transform; "" means same as the single source
	OutShape  []int32 // shape after the transform; nil means same as the single source
}

TensorSpec describes one output tensor within a blob: the source tensor(s) it is built from, the transform that combines or converts them, the name it takes in the blob, and an optional quantization to apply. When Quantize is set the writer runs MLX quantization, which generates the tensor's scale and bias sub-tensors; otherwise the (transformed) bytes are stored as-is.

type Transform added in v0.31.2 

type Transform string

Transform names how a tensor's source(s) are turned into the output tensor. The zero value, TransformNone, copies a single source through unchanged. 

const (
	TransformNone Transform = ""

	// TransformRepackFP4 reinterprets a U8 fp4-packed weight (2 values/byte)
	// as U32 words (8 values/word): the bytes are unchanged, only the dtype
	// and last dimension are relabeled.
	TransformRepackFP4 Transform = "repack_fp4"

	// TransformRelabelU8 relabels an F8_E4M3 scale as U8 so the loader reads
	// its raw bytes; the bytes themselves are unchanged.
	TransformRelabelU8 Transform = "relabel_u8"

	// TransformScalarF32 validates that the source is a scalar F32 and copies
	// it through (a global scale stored as-is).
	TransformScalarF32 Transform = "scalar_f32"

	// TransformReciprocalF32 validates a scalar F32 and stores its reciprocal
	// (a global scale the producer stored inverted).
	TransformReciprocalF32 Transform = "reciprocal_f32"

	// TransformStackExperts concatenates N per-expert source tensors (in
	// expert-index order) into one [experts, ...] tensor.
	TransformStackExperts Transform = "stack_experts"

	// TransformDecodeFP8 dequantizes a block-FP8 weight using its block scale.
	// Its two sources are the F8_E4M3 weight and its scale companion; the
	// result is a BF16 tensor, which Quantize (if set) then re-quantizes.
	TransformDecodeFP8 Transform = "decode_fp8"

	// TransformDecodeStackFP8 stacks N per-expert block-FP8 weights (and their N
	// block scales) into one [experts, out, in] tensor and dequantizes it. Its
	// sources are the N weights followed by the N scales, in expert-index order;
	// the result is a BF16 tensor, which Quantize (if set) then re-quantizes.
	TransformDecodeStackFP8 Transform = "decode_stack_fp8"
)

Source Files

View all Source files

Directories

Path Synopsis
Package client provides local and server-backed model creation for safetensors-based models.
 Package client provides local and server-backed model creation for safetensors-based models.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.