Documentation
¶
Overview ¶
Package models is the engine's read-only model mount (M7 of the LLM plan, go-wasmcontainers/gpu docs/llm-design.md): a container that asks for a model finds it at /models/NAME.gguf, read-only and range-readable, and can neither write to it nor name a URL.
A model is a registry entry pinned by Hugging Face commit and sha256. The bytes come from a Source (natively a file of the host's cache, in the browser the OPFS cache filled from the pinned URL); before the first use the source's size is checked against the entry and against MaxSize, and its whole content against the entry's sha256. Only then is it mounted, as an external file of a frozen rootfs tree (rootfs.FS.AddExternal), so nothing is copied into memory.
Index ¶
Constants ¶
const HashChunk = 16 << 20
HashChunk is the size of the reads that verify a source.
const MaxSize int64 = 2 << 30
MaxSize bounds a model file: the 1.5 GiB GPU grant ceiling, plus the room a GGUF header and the unquantised tensors need.
Variables ¶
var ( ErrUnknown = errors.New("models: unknown model") ErrSize = errors.New("models: wrong or excessive size") ErrChecksum = errors.New("models: sha256 mismatch") ErrHost = errors.New("models: host not allowed") )
Errors.
var Hosts = []string{"huggingface.co", "us.aws.cdn.hf.co"}
Hosts are the only hosts a model may be fetched from: Hugging Face and the CDN its resolve URLs redirect to (the playground's CSP connect-src lists the same two).
var Registry = []Entry{
{Name: "smollm2-135m-instruct-q4_0", Repo: "bartowski/SmolLM2-135M-Instruct-GGUF",
Revision: "09816acd5d99df7be770d85ea30822623dab342c", File: "SmolLM2-135M-Instruct-Q4_0.gguf",
Size: 91893088, SHA256: "bcc3af2849ad6095af57e9b5cd43775256efdc66e306acb529172f92d0c04b03", License: "apache-2.0"},
}
Registry is the list of pinned models. Entries are added by pull request, each with its commit and the sha256 read from the repository's LFS pointer.
Functions ¶
func CachePath ¶
CachePath is where an entry lives in dir: named by its sha256, so a file can never be mistaken for another model.
Types ¶
type Entry ¶
type Entry struct {
Name string // the mount name: /models/Name.gguf
Repo string // Hugging Face repository, "owner/name"
Revision string // commit, never a branch
File string // file in the repository
Size int64
SHA256 string // lowercase hex of the file (the LFS oid)
License string
}
Entry is one pinned model.
type Fetcher ¶
type Fetcher struct {
Client *http.Client // nil: http.DefaultClient with the host check
Hosts []string // nil: Hosts
}
Fetcher downloads entries into a cache directory.
func (Fetcher) Fetch ¶
Fetch downloads e from its pinned URL into dir, refusing a host outside Hosts at the first request and at every redirect, a Content-Length above the entry's size, and a body that is longer than, or hashes differently from, the entry. The file appears in dir only once verified.
type FileSource ¶
type FileSource struct {
// contains filtered or unexported fields
}
FileSource is a Source over a host file.
func OpenFile ¶
func OpenFile(path string) (*FileSource, error)
OpenFile opens a host file as a Source.
type HTTPSource ¶
type HTTPSource struct {
URL string
Client *http.Client
Hosts []string // nil: Hosts
// ChunkSize is the range read size; zero means 16 MiB (HashChunk).
ChunkSize int64
// contains filtered or unexported fields
}
HTTPSource is a Source over a URL that honours Range requests, read in aligned chunks of ChunkSize (one chunk is kept), as the browser engine reads a model from the pinned Hugging Face URL. It follows redirects only to Hosts, never reads past the entry's size, and refuses a server that answers a range with the wrong length or a whole 200 body.
func NewHTTPSource ¶
func NewHTTPSource(rawURL string, size int64, hosts []string, client *http.Client) (*HTTPSource, error)
NewHTTPSource returns a source of the given size (from the registry; the server is never trusted for it). The URL's host must be allowed.
Source Files