Documentation
¶
Overview ¶
Package warm implements the outpost's adaptive, considerate, always-on warm-serving plane. It keeps a small, conservative set of LLM models resident (zero cold-start) but yields the machine to the user's own work the moment they get busy — unloading warm models when the system-load profiler reports Busy() and restoring them when the host goes idle again.
Two halves:
- Executor.Apply — the control-endpoint action driven by cloudbox over the matrix tunnel (POST /admin/warm): load a model with a persistent keep-alive, form a shard for a too-big model, or unload a model. Each mode respects the live warm budget and is idempotent.
- Executor.RunSupervisor — the considerate yield loop. It tracks the DESIRED warm set (what cloudbox last asked to keep warm, persisted) and, on every tick, unloads that set while the host is busy and restores it (within the current warm budget) once the host is idle. So even mid-request the host protects the user's other work and self-heals when quiet.
The executor talks to the local Ollama daemon and (optionally) the shard manager through narrow interfaces so it stays testable and free of import cycles.
Index ¶
Constants ¶
const ( ModeLoad = "load" ModeShard = "shard" ModeUnload = "unload" )
Warm request modes.
const ( StatusLoaded = "loaded" StatusAlreadyResident = "already_resident" StatusShardStarted = "shard_started" StatusAlreadyActive = "already_active" StatusUnloaded = "unloaded" StatusSkippedBusy = "skipped_busy" StatusOverBudget = "over_budget" )
Response status values.
Variables ¶
This section is empty.
Functions ¶
func MountRoute ¶
func MountRoute(rg *gin.RouterGroup, e *Executor)
MountRoute attaches the cloudbox-driven warm-serving control handler at `POST /admin/warm` on rg.
Auth model mirrors `/admin/upgrade`: **no bearer at the HTTP layer.** The route lives on the daemon's matrix-tunnel-fronted main HTTP server, which binds 127.0.0.1 only, so cloudbox (through the tunnel) is the only reachable caller. The route is mounted solely on paired hosts (the caller gates on fc.AccessToken != "", same as the upgrade route). What the handler actually does is further gated by the live warm budget and the busy verdict, so even a spurious call can't override the considerate policy.
Body: WarmRequest{model, mode} where mode ∈ load | shard | unload. Reply: WarmResponse{status, active_model, busy, warm_budget_bytes}.
Types ¶
type APIError ¶
APIError carries an HTTP-style status alongside the message so the route layer can map it to a gin status code.
type Config ¶
type Config struct {
Ollama OllamaControl
Shard ShardControl
Gauge LoadGauge
UsableMem func() uint64
APIPort int // shard API port (0 → 11434)
// Desired seeds the persisted desired warm set (FileConfig.WarmDesired).
Desired []string
// PersistDesired persists the updated desired set (threaded through
// admincore so writes serialize against the config mutex). nil → the
// desired set is in-memory only (still survives within a daemon run).
PersistDesired func([]string) error
Interval time.Duration // supervisor tick (0 → 30s)
Logger *slog.Logger
}
Config wires the executor. Ollama + Gauge are required; Shard is optional. UsableMem returns the host's usable memory in bytes.
type Executor ¶
type Executor struct {
// contains filtered or unexported fields
}
Executor is the warm-serving actor: the control-endpoint handler plus the considerate yield/restore supervisor.
func (*Executor) Apply ¶
func (e *Executor) Apply(ctx context.Context, req WarmRequest) (WarmResponse, error)
Apply runs one warm request. Returns an *APIError for client-visible failures (bad mode, sharding unavailable) so the HTTP layer can map the status code; a plain error is a 500.
type OllamaControl ¶
type OllamaControl interface {
// EnsureResident makes the model resident with a persistent
// keep-alive (keep_alive: -1). When pull is true a missing model is
// pulled first; when false a missing model is a no-op error (the
// supervisor never blocks a tick on a multi-GB download).
EnsureResident(ctx context.Context, model string, pull bool) error
// Release unloads the model (keep_alive: 0). Idempotent.
Release(ctx context.Context, model string) error
// ModelSizeBytes returns the model's on-disk size (0 when unknown).
ModelSizeBytes(ctx context.Context, model string) (uint64, error)
// LoadedModels returns the names currently resident (/api/ps).
LoadedModels(ctx context.Context) ([]string, error)
// OnDisk reports whether the model is already downloaded (/api/tags).
OnDisk(ctx context.Context, model string) bool
}
OllamaControl is the subset of local-Ollama operations the executor needs. Implemented by ollamaClient (below) against the daemon's HTTP API; stubbed in tests.
func NewOllamaClient ¶
func NewOllamaClient(baseURL string) OllamaControl
NewOllamaClient builds an OllamaControl targeting baseURL.
type ShardControl ¶
type ShardControl interface {
ActiveModel() string
Orchestrate(ctx context.Context, model string, apiPort int, extra []string) error
Stop()
}
ShardControl is the subset of the shard manager the executor drives. Optional — nil disables shard mode.
type WarmRequest ¶
WarmRequest is the POST /admin/warm body. Mode is load|shard|unload.
type WarmResponse ¶
type WarmResponse struct {
Status string `json:"status"`
ActiveModel string `json:"active_model,omitempty"`
Busy bool `json:"busy"`
WarmBudgetBytes int64 `json:"warm_budget_bytes"`
}
WarmResponse is the endpoint's reply. ActiveModel is the model this host is currently serving via a shard (or the model just warmed); Busy + WarmBudgetBytes echo the live load verdict so the caller can see why a load/shard was skipped.