Documentation
¶
Overview ¶
Package clusterllm is outpost's passive integrator for an intra-home distributed-inference backend — a runtime that tensor/pipeline-splits a single model across several member machines so a home can serve a model too large for any one box. It mirrors the builtin_apps DetectOllama / DetectPodman shape exactly: HTTP-probe only, never spawn, never manage a lifecycle. Whatever started the backend (the operator's `podman run` against the ycode-published socket, a future vkpodman-scheduled Pod, anything else) is irrelevant here.
The first and only driver today is GPUStack (Apache-2.0, OpenAI-compatible, heterogeneous NVIDIA/AMD/Apple/Ascend). The package is written backend-agnostic on purpose: a distributed-llama or native-ycode backend that publishes the same reachability + worker shape drops in by adding a driver and a Backend string, with no change to the outpost↔ cloudbox wire seam (the ollama registry push) or to cloudbox's tier-0 router.
Safety contract: every failure mode (endpoint down, wrong path, missing API key, schema drift) degrades to an inert result — State NotReachable, or Running with MaxModelBytes 0 — so cloudbox's size filter stays off and never hides a model that would otherwise route to this host. The backend only ever *adds* reach; it can never subtract it by misbehaving.
Index ¶
Constants ¶
const (
BackendGPUStack = "gpustack"
)
Backend names. The default driver is GPUStack; the string is carried through to cloudbox on the registry push so a future swap needs no cloudbox-side re-migration.
const BackendLlamaCPP = "llamacpp"
BackendLlamaCPP is the second driver: a llama.cpp RPC shard — one leader `llama-server --rpc <worker IPs>` pipelining tensor work to N `rpc-server` workers (the topology `outpost cluster shard-init` scaffolds). It is the vk-ollama "model bigger than any one box's VRAM" path: several LAN boxes cooperatively serve one model.
Unlike GPUStack the shard exposes no Bearer-gated management API to sum worker VRAM from, so MaxModelBytes stays unknown (0, filter inert) when a shard leader is detected. What the seam *does* carry is the serving endpoint + the llamacpp backend tag, which is exactly what's needed for cloudbox's tier-0 router to discover the home and route to it; the size filter (clusterCanHold) is proprietary + separate and treats an unknown MaxModelBytes as unconstrained, so the cluster is never hidden.
const DefaultEndpoint = "http://127.0.0.1:18080"
DefaultEndpoint is the conventional loopback port operators publish a GPUStack container on. Only a default for documentation/UI hints — the operator picks the real host port when launching the container.
Variables ¶
This section is empty.
Functions ¶
func ShardEndpoint ¶ added in v0.10.0
ShardEndpoint returns the loopback serving URL a llama.cpp shard leader listens on for the given OpenAI/Ollama API port. This is the value to wire into cluster_llm_endpoint on the node running the leader so the outpost there detects the shard and advertises it through the existing seam. `outpost cluster shard-init` prints this as a formation-time advisory.
Types ¶
type Config ¶
type Config struct {
Endpoint string
APIKey string
Backend string // optional override; "" ⇒ BackendGPUStack
}
Config is the operator-supplied wiring, sourced from FileConfig (cluster_llm_endpoint / cluster_llm_api_key). Empty Endpoint disables detection. APIKey is optional: without it the backend is still detected as Running, but the worker/VRAM aggregation that powers MaxModelBytes needs the key (GPUStack's management API is Bearer-gated), so the cloudbox size filter stays inert until a key is provided.
type Detector ¶
type Detector struct {
// contains filtered or unexported fields
}
Detector caches Detect for a short TTL so the registry-push tick and the /apps poll don't each probe the backend on every cycle. Mirrors agent.BuiltinDetector. A zero/empty-endpoint Config makes Info() a cheap constant (StateUnconfigured) with no network calls.
func NewDetector ¶
NewDetector returns a detector over cfg with the given probe-result TTL (0 ⇒ defaultTTL). client may be nil (http.DefaultClient).
type Info ¶
type Info struct {
Backend string `json:"backend,omitempty"`
State State `json:"state"`
Endpoint string `json:"endpoint,omitempty"`
Version string `json:"version,omitempty"`
MemberCount int `json:"member_count,omitempty"`
AggregateVRAMBytes uint64 `json:"aggregate_vram_bytes,omitempty"`
Models []ModelInfo `json:"models,omitempty"`
}
Info is the detection result. AggregateVRAMBytes is the summed total memory across all member GPU devices (0 when unknown); the watcher maps it onto the registry push's ClusterCapacity.MaxModelBytes, and the admin UI renders the whole struct.
type ModelInfo ¶ added in v0.10.0
ModelInfo is one model served by an OpenAI-compatible cluster backend. It maps onto ollama.ModelInfo at the registry-push edge without coupling this package to the ollama wire types.
type State ¶
type State string
State is the coarse health of the configured backend.
const ( // StateUnconfigured: no endpoint set — detection is off entirely. StateUnconfigured State = "unconfigured" // StateRunning: the endpoint answered an HTTP probe (any status, // including 401 — auth-required still proves a daemon is listening). StateRunning State = "running" // StateNotReachable: the endpoint is configured but nothing answered. StateNotReachable State = "not_reachable" )
Source Files