Documentation
¶
Overview ¶
Package models is the engine's read-only model mount (M7 of the LLM plan, go-wasmcontainers/gpu docs/llm-design.md): a container that asks for a model finds it at /models/NAME.gguf, read-only and range-readable, and can neither write to it nor name a URL.
A model is a registry entry pinned by Hugging Face commit and sha256. The bytes come from a Source (natively a file of the host's cache, in the browser the OPFS cache filled from the pinned URL); before the first use the source's size is checked against the entry and against MaxSize, and its whole content against the entry's sha256. Only then is it mounted, as an external file of a frozen rootfs tree (rootfs.FS.AddExternal), so nothing is copied into memory.
Index ¶
- Constants
- Variables
- func CacheDir() (string, error)
- func CachePath(dir string, e Entry) string
- func Mount(srcs map[string]Source, entries []Entry) (*rootfs.FS, error)
- func ReportProgress(ctx context.Context, e Entry, done, total int64)
- func Verify(src Source, e Entry) error
- func WithProgress(ctx context.Context, fn ProgressFunc) context.Context
- type Cached
- type Entry
- type Fetcher
- type FileSource
- type HTTPSource
- type ProgressFunc
- type Source
Constants ¶
const HashChunk = 16 << 20
HashChunk is the size of the reads that verify a source.
const MaxSize int64 = 2 << 30
MaxSize bounds a model file: the 1.5 GiB GPU grant ceiling, plus the room a GGUF header and the unquantised tensors need.
Variables ¶
var ( ErrUnknown = errors.New("models: unknown model") ErrSize = errors.New("models: wrong or excessive size") ErrChecksum = errors.New("models: sha256 mismatch") ErrHost = errors.New("models: host not allowed") )
Errors.
var Hosts = []string{"huggingface.co", "us.aws.cdn.hf.co"}
Hosts are the only hosts a model may be fetched from: Hugging Face and the CDN its resolve URLs redirect to (the playground's CSP connect-src lists the same two).
var Registry = []Entry{
{Name: "smollm2-135m-instruct-q4_0", Repo: "bartowski/SmolLM2-135M-Instruct-GGUF",
Revision: "09816acd5d99df7be770d85ea30822623dab342c", File: "SmolLM2-135M-Instruct-Q4_0.gguf",
Size: 91893088, SHA256: "bcc3af2849ad6095af57e9b5cd43775256efdc66e306acb529172f92d0c04b03", License: "apache-2.0"},
}
Registry is the list of pinned models. Entries are added by pull request, each with its commit and the sha256 read from the repository's LFS pointer.
Functions ¶
func CachePath ¶
CachePath is where an entry lives in dir: named by its sha256, so a file can never be mistaken for another model.
func Mount ¶
Mount builds the tree mounted at /models: one external, read-only file per verified source. The tree is frozen. Every source must already be verified (use Open or Verify): Mount does not read it.
func ReportProgress ¶ added in v0.35.0
ReportProgress calls the ProgressFunc of ctx, if any.
func Verify ¶
Verify checks a source against an entry before it is used: the size first (so nothing is read from a source of the wrong or an excessive size), then the sha256 of the whole content, read in HashChunk pieces.
func WithProgress ¶ added in v0.35.0
func WithProgress(ctx context.Context, fn ProgressFunc) context.Context
WithProgress returns a context whose downloads (OpenOPFS) report to fn.
Types ¶
type Cached ¶ added in v0.35.0
type Cached struct {
Name string // the registry name, or "" for a file the registry does not list
File string // the file in the cache: SHA256.gguf
Size int64
Pinned bool // the registry lists it
}
Cached is one model file of the browser cache.
type Entry ¶
type Entry struct {
Name string // the mount name: /models/Name.gguf
Repo string // Hugging Face repository, "owner/name"
Revision string // commit, never a branch
File string // file in the repository
Size int64
SHA256 string // lowercase hex of the file (the LFS oid)
License string
}
Entry is one pinned model.
type Fetcher ¶
type Fetcher struct {
Client *http.Client // nil: http.DefaultClient with the host check
Hosts []string // nil: Hosts
}
Fetcher downloads entries into a cache directory.
func (Fetcher) Fetch ¶
Fetch downloads e from its pinned URL into dir, refusing a host outside Hosts at the first request and at every redirect, a Content-Length above the entry's size, and a body that is longer than, or hashes differently from, the entry. The file appears in dir only once verified.
type FileSource ¶
type FileSource struct {
// contains filtered or unexported fields
}
FileSource is a Source over a host file.
func OpenFile ¶
func OpenFile(path string) (*FileSource, error)
OpenFile opens a host file as a Source.
type HTTPSource ¶
type HTTPSource struct {
URL string
Client *http.Client
Hosts []string // nil: Hosts
// ChunkSize is the range read size; zero means 16 MiB (HashChunk).
ChunkSize int64
// contains filtered or unexported fields
}
HTTPSource is a Source over a URL that honours Range requests, read in aligned chunks of ChunkSize (one chunk is kept), as the browser engine reads a model from the pinned Hugging Face URL. It follows redirects only to Hosts, never reads past the entry's size, and refuses a server that answers a range with the wrong length or a whole 200 body.
func NewHTTPSource ¶
func NewHTTPSource(rawURL string, size int64, hosts []string, client *http.Client) (*HTTPSource, error)
NewHTTPSource returns a source of the given size (from the registry; the server is never trusted for it). The URL's host must be allowed.
type ProgressFunc ¶ added in v0.35.0
ProgressFunc is told how many bytes of a model are stored so far, and the total (Entry.Size), while a download runs. It is called from the downloading goroutine, never concurrently, at most once per chunk (HashChunk) and once at the end (done == total).
Source Files