warm

package
v0.14.9-dev Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 25, 2026 License: MIT Imports: 13 Imported by: 0

Documentation

Overview

Package warm implements the outpost's adaptive, considerate, always-on warm-serving plane. It keeps a small, conservative set of LLM models resident (zero cold-start) but yields the machine to the user's own work the moment they get busy — unloading warm models when the system-load profiler reports Busy() and restoring them when the host goes idle again.

Two halves:

  • Executor.Apply — the control-endpoint action driven by cloudbox over the matrix tunnel (POST /admin/warm): load a model with a persistent keep-alive, form a shard for a too-big model, or unload a model. Each mode respects the live warm budget and is idempotent.
  • Executor.RunSupervisor — the considerate yield loop. It tracks the DESIRED warm set (what cloudbox last asked to keep warm, persisted) and, on every tick, unloads that set while the host is busy and restores it (within the current warm budget) once the host is idle. So even mid-request the host protects the user's other work and self-heals when quiet.

The executor talks to the local Ollama daemon and (optionally) the shard manager through narrow interfaces so it stays testable and free of import cycles.

Index

Constants

View Source
const (
	ModeLoad   = "load"
	ModeShard  = "shard"
	ModeUnload = "unload"
)

Warm request modes.

View Source
const (
	StatusLoaded          = "loaded"
	StatusAlreadyResident = "already_resident"
	StatusShardStarted    = "shard_started"
	StatusAlreadyActive   = "already_active"
	StatusUnloaded        = "unloaded"
	StatusSkippedBusy     = "skipped_busy"
	StatusOverBudget      = "over_budget"
)

Response status values.

Variables

This section is empty.

Functions

func MountRoute

func MountRoute(rg *gin.RouterGroup, e *Executor)

MountRoute attaches the cloudbox-driven warm-serving control handler at `POST /admin/warm` on rg.

Auth model mirrors `/admin/upgrade`: **no bearer at the HTTP layer.** The route lives on the daemon's matrix-tunnel-fronted main HTTP server, which binds 127.0.0.1 only, so cloudbox (through the tunnel) is the only reachable caller. The route is mounted solely on paired hosts (the caller gates on fc.AccessToken != "", same as the upgrade route). What the handler actually does is further gated by the live warm budget and the busy verdict, so even a spurious call can't override the considerate policy.

Body: WarmRequest{model, mode} where mode ∈ load | shard | unload. Reply: WarmResponse{status, active_model, busy, warm_budget_bytes}.

Types

type APIError

type APIError struct {
	Status int
	Msg    string
}

APIError carries an HTTP-style status alongside the message so the route layer can map it to a gin status code.

func (*APIError) Error

func (e *APIError) Error() string

type Config

type Config struct {
	Ollama    OllamaControl
	Shard     ShardControl
	Gauge     LoadGauge
	UsableMem func() uint64
	APIPort   int // shard API port (0 → 11434)

	// Desired seeds the persisted desired warm set (FileConfig.WarmDesired).
	Desired []string
	// PersistDesired persists the updated desired set (threaded through
	// admincore so writes serialize against the config mutex). nil → the
	// desired set is in-memory only (still survives within a daemon run).
	PersistDesired func([]string) error

	Interval time.Duration // supervisor tick (0 → 30s)
	Logger   *slog.Logger
}

Config wires the executor. Ollama + Gauge are required; Shard is optional. UsableMem returns the host's usable memory in bytes.

type Executor

type Executor struct {
	// contains filtered or unexported fields
}

Executor is the warm-serving actor: the control-endpoint handler plus the considerate yield/restore supervisor.

func New

func New(cfg Config) *Executor

New builds an Executor, seeding the desired set from cfg.Desired.

func (*Executor) Apply

func (e *Executor) Apply(ctx context.Context, req WarmRequest) (WarmResponse, error)

Apply runs one warm request. Returns an *APIError for client-visible failures (bad mode, sharding unavailable) so the HTTP layer can map the status code; a plain error is a 500.

func (*Executor) Desired

func (e *Executor) Desired() []string

Desired returns a sorted copy of the current desired warm set.

func (*Executor) RunSupervisor

func (e *Executor) RunSupervisor(ctx context.Context) error

RunSupervisor is the considerate yield loop. It ticks on Interval, unloading the desired warm set while the host is busy and restoring it (within the current budget) while idle. Always returns nil.

type LoadGauge

type LoadGauge interface {
	Busy() bool
	WarmBudgetBytes(usableMem uint64) int64
}

LoadGauge is the sysload profiler's warm-serving verdict.

type OllamaControl

type OllamaControl interface {
	// EnsureResident makes the model resident with a persistent
	// keep-alive (keep_alive: -1). When pull is true a missing model is
	// pulled first; when false a missing model is a no-op error (the
	// supervisor never blocks a tick on a multi-GB download).
	EnsureResident(ctx context.Context, model string, pull bool) error
	// Release unloads the model (keep_alive: 0). Idempotent.
	Release(ctx context.Context, model string) error
	// ModelSizeBytes returns the model's on-disk size (0 when unknown).
	ModelSizeBytes(ctx context.Context, model string) (uint64, error)
	// LoadedModels returns the names currently resident (/api/ps).
	LoadedModels(ctx context.Context) ([]string, error)
	// OnDisk reports whether the model is already downloaded (/api/tags).
	OnDisk(ctx context.Context, model string) bool
}

OllamaControl is the subset of local-Ollama operations the executor needs. Implemented by ollamaClient (below) against the daemon's HTTP API; stubbed in tests.

func NewOllamaClient

func NewOllamaClient(baseURL string) OllamaControl

NewOllamaClient builds an OllamaControl targeting baseURL.

type ShardControl

type ShardControl interface {
	ActiveModel() string
	Orchestrate(ctx context.Context, model string, apiPort int, extra []string) error
	Stop()
}

ShardControl is the subset of the shard manager the executor drives. Optional — nil disables shard mode.

type WarmRequest

type WarmRequest struct {
	Model string `json:"model"`
	Mode  string `json:"mode"`
}

WarmRequest is the POST /admin/warm body. Mode is load|shard|unload.

type WarmResponse

type WarmResponse struct {
	Status          string `json:"status"`
	ActiveModel     string `json:"active_model,omitempty"`
	Busy            bool   `json:"busy"`
	WarmBudgetBytes int64  `json:"warm_budget_bytes"`
}

WarmResponse is the endpoint's reply. ActiveModel is the model this host is currently serving via a shard (or the model just warmed); Busy + WarmBudgetBytes echo the live load verdict so the caller can see why a load/shard was skipped.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL