localmodel

package
v0.21.5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 1, 2026 License: Apache-2.0 Imports: 20 Imported by: 0

Documentation

Overview

Model download + on-disk config helpers (split from model.go). EnsureModel fetches a model when missing (with progress + download polling), patchTokenizerConfig normalizes a freshly-downloaded tokenizer, and the dir-level guards (dirSizeBytes, hasModelWeights, validModelConfig) decide whether a model directory is usable. The catalog / RAM-tiering / resolution logic stays in model.go.

Package localmodel manages the lifecycle of the local LLM server (cmd/llm_server) that runs on Apple Silicon via MLX. It handles:

  • Detecting whether a server is already running on the configured port
  • Spawning a new server as a detached background process
  • Downloading models from HuggingFace with progress reporting
  • Selecting the best model for the machine's RAM

The package is the glue between sprout's provider system (which treats sprout-local as just another OpenAI-compatible endpoint) and the actual Go MLX server binary. When a user selects the "Local" provider in onboarding, this package ensures the server is running with the right model before the provider is used.

Index

Constants

View Source 
const DefaultPort = 18081

DefaultPort is the port the local LLM server listens on (for the standalone HTTP server mode). The in-process provider doesn't use it.

Variables

View Source 
var DefaultModelsDir = resolveDefaultModelsDir()

DefaultModelsDir is where downloaded LLM (chat) model weights are stored — ~/.sprout-local/models. $SPROUT_LLM_MODELS_DIR overrides for tests and custom layouts.

History: ~/dev/llm-models was the original hardcoded default (a personal dev-machine convention that leaked into the product), superseded by an XDG data-dir layout, and finally by ~/.sprout-local/models — a dedicated, predictable root users can find and size at a glance, with one subdirectory per model. Installations with weights at a previous location are migrated once, at first resolve: each model directory is renamed into the new root (same-volume renames are instant); a cross-volume move falls back to copy+delete; anything that can't move is left in place and keeps resolving to the old root until it can be migrated (partial migration is safe — remaining dirs migrate on a later attempt).

Functions

func CancelDownload added in v0.18.0 

func CancelDownload(modelID string) error

CancelDownload cancels a running download for the given model.

func DisableForTesting added in v0.18.9 

func DisableForTesting()

DisableForTesting is a no-op on platforms without MLX: the stub provider never loads model weights, so there is nothing to pin off. It exists to mirror the Apple Silicon method (see local_provider.go) so callers can compile on every platform.

func DownloadModelsDir added in v0.18.0 

func DownloadModelsDir() string

DownloadModelsDir exposes the models root for API responses (the webui shows users where weights live).

func EnsureModel 

func EnsureModel(ctx context.Context, status ModelStatus, progressFn ProgressCallback) (string, error)

EnsureModel downloads a model if it's not already installed.

func EnsureServerForProvider 

func EnsureServerForProvider(ctx context.Context, providerID string) error

EnsureServerForProvider ensures the local model is loaded and ready. On Apple Silicon with MLX, and on Linux ARM64 with the GGML backend, this loads the model in-process (direct compute call, no HTTP server). On other platforms it returns an error.

Safe to call on every request — short-circuits when the model is already loaded.

func EnsureServerForProviderWithCheck 

func EnsureServerForProviderWithCheck(ctx context.Context, providerID string) error

EnsureServerForProviderWithCheck loads the local model in-process. No HTTP server is spawned — the model runs directly via MLX. This is the proactive pre-load path called when the user switches to sprout-local. Returns nil if the model is already loaded.

Equivalent to EnsureServerForProviderWithCheckAndModel with no model hint — auto-selects by RAM tier. Kept as a separate name since most callers (the local-provider recovery hook, onboarding) have no specific model in mind and auto-selection is exactly right for them.

func EnsureServerForProviderWithCheckAndModel 

func EnsureServerForProviderWithCheckAndModel(ctx context.Context, providerID, model string) error

EnsureServerForProviderWithCheckAndModel is EnsureServerForProviderWithCheck, but preloads the given model (a catalog Name or installed directory basename — see ResolveModelID) instead of the RAM-tier auto-selected one, when model is non-empty.

This matters for callers preloading BEFORE the real agent (and its config-driven model resolution) exists yet — cmd/agent_command.go's createChatAgent preloads here to warm up the model while the rest of startup proceeds, but without this, that preload had no way to know the user's persisted model choice and always auto-selected instead. If that choice differs from the RAM-tier default (e.g. the user explicitly picked the larger "stretch" tier model), the auto-selected preload loads the WRONG model, and the real agent creation moments later corrects it with SetModel + a second full reload — silently doubling startup latency with an 8+ second load nobody asked for. A bad model hint (unknown ID, stale/uninstalled choice) degrades gracefully to auto-selection, same as SetModel's other callers.

func EnsureServerHealth 

func EnsureServerHealth(ctx context.Context, modelDir string) error

EnsureServerHealth checks if the server is running and starts it if not, using the given model directory. Convenience wrapper for agent integration.

func EnsureServerHealthWithBackend 

func EnsureServerHealthWithBackend(ctx context.Context, modelDir, backend string) error

EnsureServerHealthWithBackend is like EnsureServerHealth but specifies the server backend ("gomlx" or "mlx_lm").

func HasInstalledModel 

func HasInstalledModel() bool

HasInstalledModel reports whether any model is installed locally.

func IsRunning 

func IsRunning() bool

IsRunning reports whether the local model is loaded.

func IsServerPresent 

func IsServerPresent() bool

IsServerPresent reports whether any local model backend is available. Used by provider readiness checks.

func LastActivity 

func LastActivity() time.Time

LastActivity returns the timestamp of the last request, or zero if the model has never been used.

func MLXAvailable added in v0.18.14 

func MLXAvailable() bool

MLXAvailable is false on non-MLX platforms and cgo-off builds.

func PlatformSupported 

func PlatformSupported() bool

PlatformSupported reports whether the local LLM engine is supported on this platform (Linux ARM64, or Apple Silicon).

func ResetIdleForTest 

func ResetIdleForTest()

ResetIdleForTest resets the idle reaper state.

func ServerModelDir 

func ServerModelDir() string

ServerModelDir returns the model directory of the loaded model, or empty string if no model is loaded.

func StopServer 

func StopServer(port int) error

func TieredModelInfos 

func TieredModelInfos(ram uint64) []api.ModelInfo

TieredModelInfos builds the RAM-tier catalog matrix as api.ModelInfo entries for a given RAM budget — see catalog.TieredCatalogForRAM. Every tier is included (so /model shows the full roadmap, not just what's pickable today), but only the suggested and stretch tiers carry EligibleRoles; blocked tiers get an explanatory Description and no eligible/recommended roles, matching existing picker conventions for "not really pickable". Actual selection is enforced separately in LocalProvider.SetModel — this only informs the picker.

func TotalSystemRAM 

func TotalSystemRAM() uint64

TotalSystemRAM returns this machine's physical RAM in bytes, for callers outside this package that need it for RAM-tier gate checks (e.g. the /model CLI command, deciding whether to download a selection before LocalProvider.SetModel's own gate would reject it).

func TouchActivity 

func TouchActivity()

TouchActivity records that the local model is in use, resetting the idle timer. Called before each request to the local provider.

Types

type DownloadStatusPayload added in v0.18.0 

type DownloadStatusPayload struct {
	ModelID string `json:"model_id"`
	Status  string `json:"status"` // "downloading" | "completed" | "failed" | "canceled"
	Bytes   int64  `json:"bytes_downloaded"`
	Total   int64  `json:"total_bytes,omitempty"` // 0 = unknown (see pollDownloadProgress)
	Error   string `json:"error,omitempty"`
}

DownloadStatusPayload is the JSON-facing snapshot of a download job's state. Values are copied out under the job's lock, so holders can't observe torn state.

func ActiveDownloads added in v0.18.0 

func ActiveDownloads() []DownloadStatusPayload

ActiveDownloads snapshots all running jobs.

func DownloadStatus added in v0.18.0 

func DownloadStatus(modelID string) *DownloadStatusPayload

DownloadStatus returns the current status for a model, or nil when no job exists for it.

func StartDownload added in v0.18.0 

func StartDownload(modelID string) (DownloadStatusPayload, error)

StartDownload begins downloading a catalog model by ID (catalog Name or directory basename — ResolveModelID's contract) and returns its initial status. Errors when the model is unknown, already installed, or already downloading. RAM-tier policy is NOT enforced here: an explicit user selection is a warned-but-allowed choice (UIs surface the tier), matching the CLI's SPROUT_ALLOW_OVERWEIGHT behavior for explicitly-selected models.

type LocalProvider 

type LocalProvider struct{}

LocalProvider is a stub on platforms without MLX. All methods return errors indicating local LLM is unavailable. This allows the rest of sprout to compile and run on non-Apple-Silicon platforms.

func GetLocalProvider 

func GetLocalProvider() *LocalProvider

func (*LocalProvider) CheckConnection 

func (p *LocalProvider) CheckConnection() error

func (*LocalProvider) Close 

func (p *LocalProvider) Close() error

func (*LocalProvider) GetAverageTPS 

func (p *LocalProvider) GetAverageTPS() float64

func (*LocalProvider) GetLastTPS 

func (p *LocalProvider) GetLastTPS() float64

func (*LocalProvider) GetModel 

func (p *LocalProvider) GetModel() string

func (*LocalProvider) GetModelContextLimit 

func (p *LocalProvider) GetModelContextLimit() (int, error)

func (*LocalProvider) GetProvider 

func (p *LocalProvider) GetProvider() string

func (*LocalProvider) GetTPSStats 

func (p *LocalProvider) GetTPSStats() map[string]float64

func (*LocalProvider) GetVisionModel 

func (p *LocalProvider) GetVisionModel() string

func (*LocalProvider) ListModels 

func (p *LocalProvider) ListModels(ctx context.Context) ([]api.ModelInfo, error)

func (*LocalProvider) ResetTPSStats 

func (p *LocalProvider) ResetTPSStats()

func (*LocalProvider) SendChatRequest 

func (p *LocalProvider) SendChatRequest(ctx context.Context, messages []api.Message, tools []api.Tool, reasoning string, disableThinking bool) (*api.ChatResponse, error)

func (*LocalProvider) SendChatRequestStream 

func (p *LocalProvider) SendChatRequestStream(ctx context.Context, messages []api.Message, tools []api.Tool, reasoning string, disableThinking bool, callback api.StreamCallback) (*api.ChatResponse, error)

func (*LocalProvider) SendVisionRequest 

func (p *LocalProvider) SendVisionRequest(ctx context.Context, messages []api.Message, tools []api.Tool, reasoning string, disableThinking bool) (*api.ChatResponse, error)

func (*LocalProvider) SetDebug 

func (p *LocalProvider) SetDebug(bool)

func (*LocalProvider) SetModel 

func (p *LocalProvider) SetModel(string) error

func (*LocalProvider) SupportsVision 

func (p *LocalProvider) SupportsVision() bool

func (*LocalProvider) VisionCapabilities 

func (p *LocalProvider) VisionCapabilities() api.VisionCapabilities

type ModelStatus 

type ModelStatus struct {
	Name      string `json:"name"`
	Dir       string `json:"dir"`
	HFRepo    string `json:"hf_repo"`
	HFInclude string `json:"hf_include,omitempty"`
	// MinRAM is the minimum RAM to select this model at all (with a warning
	// if below MinRAMSuggested). 0 for the entry-level model.
	MinRAM uint64 `json:"min_ram_gb"`
	// MinRAMSuggested is the RAM at which this model becomes the unwarned
	// default. Capped display-side for the top-of-line model, whose real
	// threshold is intentionally unbounded (it never becomes an unwarned
	// default — see catalog.go) rather than showing a nonsensical value.
	MinRAMSuggested uint64 `json:"min_ram_suggested_gb,omitempty"`
	Installed       bool   `json:"installed"`
	Size            int64  `json:"size_bytes"`
	IsTuned         bool   `json:"is_tuned"`
	ParamSize       string `json:"param_size"`
	QuantBits       string `json:"quant_bits"`
	ServerBackend   string `json:"server_backend,omitempty"`
}

ModelStatus describes a model from the user's perspective — either a downloadable catalog entry or an installed model discovered on disk. Thresholds here are RAM-agnostic (the raw catalog values); classifying a model as suggested/stretch/blocked for a specific machine is done separately via catalog.TieredCatalogForRAM, which needs actual RAM in hand.

func ListModels 

func ListModels() []ModelStatus

ListModels returns all models: installed variants discovered on disk (including sprout-tuned, different quant levels) plus downloadable catalog entries that aren't installed. Installed models are listed first, sorted by param size (largest first).

func RecommendedModel 

func RecommendedModel(ramBytes uint64) *ModelStatus

RecommendedModel returns the best model for the machine's RAM that is already installed, preferring sprout-tuned variants. Delegates to catalog.SelectModelForRAM so this shares the same RAM-gate and quant preference (mlx-q5 > q5 > q8 > unquantized) logic as onboarding and the standalone server — see preferTunedQuant.

func ResolveModelID 

func ResolveModelID(id string) (*ModelStatus, error)

ResolveModelID finds the ModelStatus for a stable model ID — either a catalog tier Name (e.g. "qwen3.5-9b", preferring an installed sprout-tuned variant of the same size, matching SelectModelForRAM's own preference) or an installed directory's exact basename. Returns an error if unknown.

type ProgressCallback 

type ProgressCallback func(downloaded, total int64)

ProgressCallback is called during model download with bytes downloaded and total bytes (0 if unknown).

type ServerStatus 

type ServerStatus struct {
	Running bool   `json:"running"`
	Port    int    `json:"port"`
	Model   string `json:"model"`
	Healthy bool   `json:"healthy"`
	URL     string `json:"url"`
	PID     int    `json:"pid,omitempty"`
	Error   string `json:"error,omitempty"`
}

ServerStatus describes the state of the local LLM server.

func EnsureServer 

func EnsureServer(ctx context.Context, port int, modelDir string) (*ServerStatus, error)

EnsureServer checks if the local LLM server is running and healthy on the given port. If not, it spawns a new server with the given model directory. Returns the server status when the server is healthy, or an error if it couldn't be started within the timeout.

The modelDir must be an absolute path to a directory containing a valid model (config.json, tokenizer.json, *.safetensors).

func EnsureServerWithBackend 

func EnsureServerWithBackend(ctx context.Context, port int, modelDir, backend string) (*ServerStatus, error)

EnsureServerWithBackend is like EnsureServer but lets the caller specify which server backend to use: "gomlx" (default, Go-native) or "mlx_lm" (Python mlx_lm.server, needed for models without a Go architecture).

func HealthCheck 

func HealthCheck(port int) (*ServerStatus, error)

HealthCheck pings the server's /health endpoint. Returns nil if the server is healthy and running. Returns a non-nil error otherwise.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.