localmodel

package
v0.17.18 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 24, 2026 License: MIT Imports: 19 Imported by: 0

Documentation

Overview

Package localmodel manages the lifecycle of the local LLM server (cmd/llm_server) that runs on Apple Silicon via MLX. It handles:

  • Detecting whether a server is already running on the configured port
  • Spawning a new server as a detached background process
  • Downloading models from HuggingFace with progress reporting
  • Selecting the best model for the machine's RAM

The package is the glue between sprout's provider system (which treats sprout-local as just another OpenAI-compatible endpoint) and the actual Go MLX server binary. When a user selects the "Local" provider in onboarding, this package ensures the server is running with the right model before the provider is used.

Index

Constants

View Source 
const DefaultPort = 18081

DefaultPort is the port the local LLM server listens on (for the standalone HTTP server mode). The in-process provider doesn't use it.

Variables

View Source 
var DefaultModelsDir = resolveDefaultModelsDir()

DefaultModelsDir is where downloaded LLM (chat) model weights are stored — XDG-style, honoring $SPROUT_DATA_DIR/$XDG_DATA_HOME like sprout's other data, under the SAME models/ root pkg/embedding's DefaultModelDir uses (<DataDir>/models/embedding for embedding models), but its own "llm" subdirectory — one shared, discoverable "models" root with a subdirectory per kind, rather than two same-purpose-sounding but unrelated top-level directories under DataDir. $SPROUT_MODELS_DIR is already pkg/embedding's env var for its own directory, so this uses a distinct name.

~/dev/llm-models was the prior hardcoded default here: a personal dev-machine convention ("~/dev/...") baked in as if it were universal, not a real per-user default — most tools use a dot-folder or XDG path for downloaded model weights.

Resolution: $SPROUT_LLM_MODELS_DIR → $SPROUT_DATA_DIR/models/llm → $XDG_DATA_HOME/sprout/models/llm → $HOME/.local/share/sprout/models/llm. Falls back to the legacy ~/dev/llm-models location only when it already has content and the new location doesn't, so installations that downloaded models before this fix keep working without needing to re-download or manually migrate anything. New installs go straight to the proper location.

Functions

func EnsureModel 

func EnsureModel(ctx context.Context, status ModelStatus, progressFn ProgressCallback) (string, error)

EnsureModel downloads a model if it's not already installed.

func EnsureServerForProvider 

func EnsureServerForProvider(ctx context.Context, providerID string) error

EnsureServerForProvider ensures the local model is loaded and ready. On Apple Silicon with MLX, and on Linux ARM64 with the GGML backend, this loads the model in-process (direct compute call, no HTTP server). On other platforms it returns an error.

Safe to call on every request — short-circuits when the model is already loaded.

func EnsureServerForProviderWithCheck 

func EnsureServerForProviderWithCheck(ctx context.Context, providerID string) error

EnsureServerForProviderWithCheck loads the local model in-process. No HTTP server is spawned — the model runs directly via MLX. This is the proactive pre-load path called when the user switches to sprout-local. Returns nil if the model is already loaded.

Equivalent to EnsureServerForProviderWithCheckAndModel with no model hint — auto-selects by RAM tier. Kept as a separate name since most callers (the local-provider recovery hook, onboarding) have no specific model in mind and auto-selection is exactly right for them.

func EnsureServerForProviderWithCheckAndModel 

func EnsureServerForProviderWithCheckAndModel(ctx context.Context, providerID, model string) error

EnsureServerForProviderWithCheckAndModel is EnsureServerForProviderWithCheck, but preloads the given model (a catalog Name or installed directory basename — see ResolveModelID) instead of the RAM-tier auto-selected one, when model is non-empty.

This matters for callers preloading BEFORE the real agent (and its config-driven model resolution) exists yet — cmd/agent_command.go's createChatAgent preloads here to warm up the model while the rest of startup proceeds, but without this, that preload had no way to know the user's persisted model choice and always auto-selected instead. If that choice differs from the RAM-tier default (e.g. the user explicitly picked the larger "stretch" tier model), the auto-selected preload loads the WRONG model, and the real agent creation moments later corrects it with SetModel + a second full reload — silently doubling startup latency with an 8+ second load nobody asked for. A bad model hint (unknown ID, stale/uninstalled choice) degrades gracefully to auto-selection, same as SetModel's other callers.

func EnsureServerHealth 

func EnsureServerHealth(ctx context.Context, modelDir string) error

EnsureServerHealth checks if the server is running and starts it if not, using the given model directory. Convenience wrapper for agent integration.

func EnsureServerHealthWithBackend 

func EnsureServerHealthWithBackend(ctx context.Context, modelDir, backend string) error

EnsureServerHealthWithBackend is like EnsureServerHealth but specifies the server backend ("gomlx" or "mlx_lm").

func HasInstalledModel 

func HasInstalledModel() bool

HasInstalledModel reports whether any model is installed locally.

func IsRunning 

func IsRunning() bool

IsRunning reports whether the local model is loaded.

func IsServerPresent 

func IsServerPresent() bool

IsServerPresent reports whether any local model backend is available. Used by provider readiness checks.

func LastActivity 

func LastActivity() time.Time

LastActivity returns the timestamp of the last request, or zero if the model has never been used.

func PlatformSupported 

func PlatformSupported() bool

PlatformSupported reports whether the local LLM engine is supported on this platform (Linux ARM64, or Apple Silicon).

func ResetIdleForTest 

func ResetIdleForTest()

ResetIdleForTest resets the idle reaper state.

func ServerModelDir 

func ServerModelDir() string

ServerModelDir returns the model directory of the loaded model, or empty string if no model is loaded.

func StopServer 

func StopServer(port int) error

func TieredModelInfos 

func TieredModelInfos(ram uint64) []api.ModelInfo

TieredModelInfos builds the RAM-tier catalog matrix as api.ModelInfo entries for a given RAM budget — see catalog.TieredCatalogForRAM. Every tier is included (so /model shows the full roadmap, not just what's pickable today), but only the suggested and stretch tiers carry EligibleRoles; blocked tiers get an explanatory Description and no eligible/recommended roles, matching existing picker conventions for "not really pickable". Actual selection is enforced separately in LocalProvider.SetModel — this only informs the picker.

func TotalSystemRAM 

func TotalSystemRAM() uint64

TotalSystemRAM returns this machine's physical RAM in bytes, for callers outside this package that need it for RAM-tier gate checks (e.g. the /model CLI command, deciding whether to download a selection before LocalProvider.SetModel's own gate would reject it).

func TouchActivity 

func TouchActivity()

TouchActivity records that the local model is in use, resetting the idle timer. Called before each request to the local provider.

Types

type LocalProvider 

type LocalProvider struct{}

LocalProvider is a stub on platforms without MLX. All methods return errors indicating local LLM is unavailable. This allows the rest of sprout to compile and run on non-Apple-Silicon platforms.

func GetLocalProvider 

func GetLocalProvider() *LocalProvider

func (*LocalProvider) CheckConnection 

func (p *LocalProvider) CheckConnection() error

func (*LocalProvider) Close 

func (p *LocalProvider) Close() error

func (*LocalProvider) GetAverageTPS 

func (p *LocalProvider) GetAverageTPS() float64

func (*LocalProvider) GetLastTPS 

func (p *LocalProvider) GetLastTPS() float64

func (*LocalProvider) GetModel 

func (p *LocalProvider) GetModel() string

func (*LocalProvider) GetModelContextLimit 

func (p *LocalProvider) GetModelContextLimit() (int, error)

func (*LocalProvider) GetProvider 

func (p *LocalProvider) GetProvider() string

func (*LocalProvider) GetTPSStats 

func (p *LocalProvider) GetTPSStats() map[string]float64

func (*LocalProvider) GetVisionModel 

func (p *LocalProvider) GetVisionModel() string

func (*LocalProvider) ListModels 

func (p *LocalProvider) ListModels(ctx context.Context) ([]api.ModelInfo, error)

func (*LocalProvider) ResetTPSStats 

func (p *LocalProvider) ResetTPSStats()

func (*LocalProvider) SendChatRequest 

func (p *LocalProvider) SendChatRequest(ctx context.Context, messages []api.Message, tools []api.Tool, reasoning string, disableThinking bool) (*api.ChatResponse, error)

func (*LocalProvider) SendChatRequestStream 

func (p *LocalProvider) SendChatRequestStream(ctx context.Context, messages []api.Message, tools []api.Tool, reasoning string, disableThinking bool, callback api.StreamCallback) (*api.ChatResponse, error)

func (*LocalProvider) SendVisionRequest 

func (p *LocalProvider) SendVisionRequest(ctx context.Context, messages []api.Message, tools []api.Tool, reasoning string, disableThinking bool) (*api.ChatResponse, error)

func (*LocalProvider) SetDebug 

func (p *LocalProvider) SetDebug(bool)

func (*LocalProvider) SetModel 

func (p *LocalProvider) SetModel(string) error

func (*LocalProvider) SupportsConversationalVision 

func (p *LocalProvider) SupportsConversationalVision() bool

func (*LocalProvider) SupportsVision 

func (p *LocalProvider) SupportsVision() bool

func (*LocalProvider) VisionCapabilities 

func (p *LocalProvider) VisionCapabilities() api.VisionCapabilities

type ModelStatus 

type ModelStatus struct {
	Name      string `json:"name"`
	Dir       string `json:"dir"`
	HFRepo    string `json:"hf_repo"`
	HFInclude string `json:"hf_include,omitempty"`
	// MinRAM is the minimum RAM to select this model at all (with a warning
	// if below MinRAMSuggested). 0 for the entry-level model.
	MinRAM uint64 `json:"min_ram_gb"`
	// MinRAMSuggested is the RAM at which this model becomes the unwarned
	// default. Capped display-side for the top-of-line model, whose real
	// threshold is intentionally unbounded (it never becomes an unwarned
	// default — see catalog.go) rather than showing a nonsensical value.
	MinRAMSuggested uint64 `json:"min_ram_suggested_gb,omitempty"`
	Installed       bool   `json:"installed"`
	Size            int64  `json:"size_bytes"`
	IsTuned         bool   `json:"is_tuned"`
	ParamSize       string `json:"param_size"`
	QuantBits       string `json:"quant_bits"`
	ServerBackend   string `json:"server_backend,omitempty"`
}

ModelStatus describes a model from the user's perspective — either a downloadable catalog entry or an installed model discovered on disk. Thresholds here are RAM-agnostic (the raw catalog values); classifying a model as suggested/stretch/blocked for a specific machine is done separately via catalog.TieredCatalogForRAM, which needs actual RAM in hand.

func ListModels 

func ListModels() []ModelStatus

ListModels returns all models: installed variants discovered on disk (including sprout-tuned, different quant levels) plus downloadable catalog entries that aren't installed. Installed models are listed first, sorted by param size (largest first).

func RecommendedModel 

func RecommendedModel(ramBytes uint64) *ModelStatus

RecommendedModel returns the best model for the machine's RAM that is already installed, preferring sprout-tuned variants. Delegates to catalog.SelectModelForRAM so this shares the same RAM-gate and quant preference (mlx-q5 > q5 > q8 > unquantized) logic as onboarding and the standalone server — see preferTunedQuant.

func ResolveModelID 

func ResolveModelID(id string) (*ModelStatus, error)

ResolveModelID finds the ModelStatus for a stable model ID — either a catalog tier Name (e.g. "qwen3.5-9b", preferring an installed sprout-tuned variant of the same size, matching SelectModelForRAM's own preference) or an installed directory's exact basename. Returns an error if unknown.

type ProgressCallback 

type ProgressCallback func(downloaded, total int64)

ProgressCallback is called during model download with bytes downloaded and total bytes (0 if unknown).

type ServerStatus 

type ServerStatus struct {
	Running bool   `json:"running"`
	Port    int    `json:"port"`
	Model   string `json:"model"`
	Healthy bool   `json:"healthy"`
	URL     string `json:"url"`
	PID     int    `json:"pid,omitempty"`
	Error   string `json:"error,omitempty"`
}

ServerStatus describes the state of the local LLM server.

func EnsureServer 

func EnsureServer(ctx context.Context, port int, modelDir string) (*ServerStatus, error)

EnsureServer checks if the local LLM server is running and healthy on the given port. If not, it spawns a new server with the given model directory. Returns the server status when the server is healthy, or an error if it couldn't be started within the timeout.

The modelDir must be an absolute path to a directory containing a valid model (config.json, tokenizer.json, *.safetensors).

func EnsureServerWithBackend 

func EnsureServerWithBackend(ctx context.Context, port int, modelDir, backend string) (*ServerStatus, error)

EnsureServerWithBackend is like EnsureServer but lets the caller specify which server backend to use: "gomlx" (default, Go-native) or "mlx_lm" (Python mlx_lm.server, needed for models without a Go architecture).

func HealthCheck 

func HealthCheck(port int) (*ServerStatus, error)

HealthCheck pings the server's /health endpoint. Returns nil if the server is healthy and running. Returns a non-nil error otherwise.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.