Documentation
¶
Overview ¶
Package serve runs a local model server and keeps track of it. It is the step between a built launch plan (a serve command that has not been started) and a model a client can reach: it starts the runtime as a confined background process, waits for its loopback endpoint to answer, and records the running server so a later command can find, reuse, or stop it. Starting a server that is already up is a no-op, so the same call both starts and reuses.
The runtime is the code-execution surface that parses untrusted weights, so it is started inside the sandbox rather than as a bare child. The package depends on small interfaces for launching, probing, and killing, so the whole lifecycle is testable with fakes and no live runtime, while the real wiring uses the sandbox and HTTP.
Index ¶
- func EmitStats(ctx context.Context, m observe.Meter, modelID string, s RuntimeStats)
- func EngineStopper(engine, id string) error
- func OSKiller(pid int) error
- type ContainerEnsureConfig
- type ContainerStopper
- type Endpoint
- type EnsureConfig
- type Killer
- type Launcher
- type Manager
- func (m *Manager) Ensure(ctx context.Context, cfg EnsureConfig) (Endpoint, error)
- func (m *Manager) EnsureContainer(ctx context.Context, cfg ContainerEnsureConfig) (Endpoint, error)
- func (m *Manager) Stats(ctx context.Context, modelID string) (RuntimeStats, error)
- func (m *Manager) Status(ctx context.Context) ([]Record, error)
- func (m *Manager) Stop(modelID string) (bool, error)
- type NopStatsSource
- type Option
- type Prober
- type Proc
- type PromStatsSource
- type Record
- type Registry
- type RuntimeStats
- type SandboxLauncher
- type StatsSource
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func EmitStats ¶
EmitStats records a load snapshot on the meter so a host's telemetry backend sees the same numbers the control plane reads. The Meter port offers counters and histograms but no gauge instrument, so the snapshot's gauge-like values are recorded as histogram observations, which a backend renders as the per-model distribution and last value. A snapshot with Known false, or a nil meter, records nothing.
func EngineStopper ¶
EngineStopper stops a container-backed server by driving the OCI engine, the production ContainerStopper. It bounds the stop with its own timeout inside the sandbox helper, so a `models stop` for a vLLM container tears it down the same way OSKiller stops a process.
func OSKiller ¶
OSKiller stops a process by pid through the operating system, the production Killer. It is used to stop a server a previous Flynn invocation started, where only the pid survives in the registry. Killing an already-gone process is reported by the OS and surfaced to the caller, which treats "no such process" as already stopped.
Types ¶
type ContainerEnsureConfig ¶
type ContainerEnsureConfig struct {
// ModelID is the catalog id, the registry key.
ModelID string
// Runtime names the runtime, recorded for display (for example "vllm").
Runtime string
// Spec is the validated container request: the digest-pinned image, the untrusted
// guarantees, the GPU grant, the published loopback port, and the server command.
Spec sandbox.ContainerSpec
// BaseURL and Port are the loopback endpoint the served model answers on, the same
// OpenAI-compatible coordinates a process-backed server reports.
BaseURL string
Port int
// ReadyTimeout overrides how long to wait for the endpoint to come up. A container
// runtime can be far slower to first answer than a process (a GPU server loads weights,
// captures CUDA graphs, and compiles kernels on first start), so the caller raises it
// above the manager's process-oriented default. Zero uses the manager default.
ReadyTimeout time.Duration
}
ContainerEnsureConfig is the input to EnsureContainer: which model, the validated container request, and the loopback endpoint to probe. The Spec's Command is the server invocation (the vLLM serve argv), built by the caller from the launch plan.
type ContainerStopper ¶
ContainerStopper stops a container-backed server by its engine and id, the container counterpart to Killer: a container has no host pid, so a separate Flynn process stops it by driving the engine. The real stopper runs the engine CLI; a test stopper records the call. A nil stopper means container-backed records cannot be stopped across invocations, so the production wiring always supplies one.
type Endpoint ¶
type Endpoint struct {
// ModelID is the catalog id being served.
ModelID string
// BaseURL is the OpenAI-compatible endpoint a client targets.
BaseURL string
// Port is the loopback port the server listens on.
Port int
// PID is the server process id.
PID int
// Reused is true when an already-running server was adopted instead of started.
Reused bool
// contains filtered or unexported fields
}
Endpoint is a running local model server reachable over loopback.
type EnsureConfig ¶
type EnsureConfig struct {
// ModelID is the catalog id, the registry key.
ModelID string
// Runtime names the runtime, recorded for display.
Runtime string
// Plan is the built, not-yet-started serve command and its loopback address.
Plan launch.Plan
// Confine requests the sandbox kernel confinement for the runtime process, which is
// the right default since the runtime parses untrusted weights.
Confine bool
}
EnsureConfig is the input to Ensure: which model, the built serve plan, and how it is run.
type Killer ¶
Killer stops a server identified only by its pid, the case where the server was started by an earlier, now-gone Flynn process and no Proc handle is held. The real killer signals the OS process; a test killer records the pid.
type Launcher ¶
Launcher starts a server process for a serve plan. The real launcher runs it inside the sandbox; a test launcher returns a fake Proc.
type Manager ¶
type Manager struct {
// contains filtered or unexported fields
}
Manager starts, reuses, reports, and stops local model servers. Its dependencies are injected so the lifecycle is exercised without a live runtime.
func NewManager ¶
NewManager builds a Manager from its launcher, prober, killer, and registry.
func (*Manager) Ensure ¶
Ensure returns a running endpoint for the model, starting the server if one is not already up. If the registry already has a server for this model and its endpoint answers a health probe, that server is adopted (Reused). Otherwise a stale record is pruned, the server is started inside the sandbox from the plan, the manager waits for its endpoint to answer, records it, and returns it. A server that starts but never becomes ready, or that exits while starting, is stopped and reported as an error with its captured output, so a launch never leaves an orphan behind.
func (*Manager) EnsureContainer ¶
EnsureContainer returns a running endpoint for a container-backed model, the container counterpart to Ensure. It reuses an already-running server when its endpoint answers, otherwise runs the container under the tier's guarantees, waits for the endpoint to come up, records it with its container identity (so a later process can stop it), and returns it. A container that starts but never becomes ready, or exits while starting, is stopped and reported with its captured output, so a launch never leaks a container. It drives the same registry and reuse path as Ensure, so the scheduler and `models status`/`stop` see process- and container-backed servers uniformly: one supervisor, two runtime shapes.
func (*Manager) Stats ¶
Stats reads a best-effort load snapshot for a running model. It returns an error only when there is no recorded server for the model. A server that is present but cannot be read (no source for its runtime, an unreachable or hung endpoint, an unparseable reply) yields a snapshot with Known false, never an error, so the control plane can always read the load without a failing call breaking it. The reading is normalized and stamped with the manager clock.
func (*Manager) Status ¶
Status returns the recorded servers whose endpoint currently answers, pruning any record that no longer does so the report reflects reality rather than stale claims. A pruned server's process is reclaimed: a record is only written after the server became ready, so an endpoint that has stopped answering means the runtime died or wedged, and a wedged runtime left running would keep holding device memory after the manager has forgotten it. Killing it before dropping the record prevents that leak; killing an already-exited process is a no-op.
type NopStatsSource ¶
type NopStatsSource struct{}
NopStatsSource reports every runtime as unknown load. It is the default for a runtime with no registered source, so the standalone path stays zero-setup.
func (NopStatsSource) Stats ¶
func (NopStatsSource) Stats(context.Context, string) (RuntimeStats, error)
Stats implements StatsSource by always reporting an unknown load.
type Option ¶
type Option func(*Manager)
Option configures a Manager.
func WithContainerStopper ¶
func WithContainerStopper(s ContainerStopper) Option
WithContainerStopper sets how a container-backed server is stopped across invocations, so `models stop` and the scheduler's evict can tear down a vLLM container the same way they kill a process-backed server. Without one, a container record cannot be stopped by a later process, so the production wiring always supplies it.
func WithPollInterval ¶
WithPollInterval sets how often Ensure probes a starting server.
func WithReadyTimeout ¶
WithReadyTimeout sets how long Ensure waits for a started server to become ready.
func WithStatsSource ¶
func WithStatsSource(runtime string, src StatsSource) Option
WithStatsSource registers a load reader for servers of a given runtime, keyed by the runtime name recorded on each server (for example "llama.cpp"). Without one for a runtime, Stats reports the load as unknown rather than failing.
type Prober ¶
Prober reports whether the model endpoint at baseURL is answering. It returns nil once the server is ready, and an error while it is not. The real prober makes a loopback HTTP request; a test prober scripts readiness.
func HTTPProbe ¶
HTTPProbe builds a Prober that reports readiness by requesting the model list off the OpenAI-compatible endpoint. A loopback model server answers that route once it has loaded the weights and is ready to serve, so a 2xx is the readiness signal. The probe uses a short per-request timeout so a hung connection does not stall the poll loop.
type Proc ¶
type Proc interface {
PID() int
Running() bool
Output() string
Done() <-chan struct{}
Stop() error
}
Proc is the handle to a started background server the manager needs: enough to know whether it is up, read why it failed, wait for it, and stop it. *sandbox.Process satisfies it; a test supplies a fake.
type PromStatsSource ¶
type PromStatsSource struct {
// contains filtered or unexported fields
}
PromStatsSource reads a running runtime's Prometheus metrics endpoint over loopback and maps the named metric families onto a RuntimeStats snapshot. The metric names differ per runtime, so the mapping is injected: LlamaCppStatsSource and VLLMStatsSource supply the two dialects Flynn serves. The read is time-boxed and size-bounded, so a hung or hostile endpoint cannot stall or exhaust the caller.
func LlamaCppStatsSource ¶
func LlamaCppStatsSource(client *http.Client) PromStatsSource
LlamaCppStatsSource reads load stats from a llama.cpp server's metrics endpoint. A nil client gets a default with a short per-request timeout.
func VLLMStatsSource ¶
func VLLMStatsSource(client *http.Client) PromStatsSource
VLLMStatsSource reads load stats from a vLLM server's metrics endpoint. A nil client gets a default with a short per-request timeout.
func (PromStatsSource) Stats ¶
func (s PromStatsSource) Stats(ctx context.Context, baseURL string) (RuntimeStats, error)
Stats scrapes the runtime's metrics endpoint and maps it onto a snapshot. A transport error, a non-2xx reply, or a read error is returned as an error for the manager to turn into an unknown reading; a 2xx body that names none of the mapped metrics yields a snapshot with Known false.
type Record ¶
type Record struct {
// ModelID is the catalog id the server runs, the key a reuse or stop looks up.
ModelID string `json:"modelID"`
// PID is the operating-system process id, so a separate process can stop the server. It
// is set for a process-backed server (the runtime runs as a host process); a
// container-backed server has no host pid and carries ContainerID instead.
PID int `json:"pid"`
// ContainerID and Engine identify a container-backed server, so a later, separate Flynn
// process can stop it by driving the engine (a container has no host pid to signal).
// They are empty for a process-backed server. Exactly one of PID or ContainerID
// identifies a given server.
ContainerID string `json:"containerID,omitempty"`
Engine string `json:"engine,omitempty"`
// Port and BaseURL are the loopback coordinates a client targets.
Port int `json:"port"`
BaseURL string `json:"baseURL"`
// Runtime names the runtime serving the model, for display.
Runtime string `json:"runtime,omitempty"`
// StartedAt is the Unix time the server was started, for display.
StartedAt int64 `json:"startedAt,omitempty"`
}
Record is the persisted description of a model server that was started, so a later, separate Flynn process (a `models status` or `models stop` invocation) can find it, report it, reach it, or kill it. It holds only loopback coordinates and a pid; no secret is ever recorded.
type Registry ¶
type Registry struct {
// contains filtered or unexported fields
}
Registry persists the set of running model servers to a single JSON file under the data directory, keyed by model id. It is the cross-process record that lets a fresh Flynn invocation see and control servers an earlier one started. Access is guarded by a mutex within a process; the file is rewritten atomically so a concurrent reader never sees a half-written file. It does not itself decide liveness: a record is a claim that a server was started, and the manager confirms it with a health probe before trusting it, pruning a record whose server no longer answers.
func NewRegistry ¶
NewRegistry returns a registry backed by servers.json under dir (created on first write). dir is typically the data directory's run area.
func (*Registry) Delete ¶
Delete removes the record for a model id, if present, and persists the registry. Deleting an absent id is not an error.
type RuntimeStats ¶
type RuntimeStats struct {
// Known is true when a usable reading was obtained and parsed. A false Known means the
// snapshot carries no information and the load must be treated as unknown, not as zero.
Known bool
// KVCacheUsage is the fraction of the key/value cache in use, in [0,1]. It is the
// primary memory-pressure signal an eviction policy reads.
KVCacheUsage float64
// RequestsRunning is the number of requests the server is actively decoding.
RequestsRunning int
// RequestsWaiting is the number of requests queued behind the running ones.
RequestsWaiting int
// DecodeTokensPerSec is the recent generation throughput, in tokens per second.
DecodeTokensPerSec float64
// PromptTokensPerSec is the recent prefill throughput, in tokens per second.
PromptTokensPerSec float64
// CollectedAt is when the reading was taken, stamped from the manager clock so it is
// deterministic in tests and consistent across the process.
CollectedAt time.Time
}
RuntimeStats is a point-in-time, best-effort reading of how loaded a running model server is: how full its key/value cache is and how many requests it is working on. It is the control-plane view the orchestrator and router read to make eviction and routing decisions, kept distinct from the metrics pushed to a backend through observe.Meter.
Fields are best-effort. A runtime that does not report a value leaves it at its zero, and Known reports whether any usable reading was obtained at all: a false Known means the snapshot carries no information (the runtime was unreachable, hung, or spoke an unparseable dialect) and a reader must treat the load as unknown rather than as zero.
type SandboxLauncher ¶
type SandboxLauncher struct {
// SB is the sandbox the server runs inside. Its working directory is where the
// runtime is confined and its environment is the deny-by-default baseline.
SB *sandbox.Local
}
SandboxLauncher starts a server process inside a Local sandbox, the production Launcher. The runtime parses untrusted weights, so running it through the sandbox rather than as a bare child is the point: confinement is applied per the Local's configuration when the serve spec requests it.
type StatsSource ¶
type StatsSource interface {
Stats(ctx context.Context, baseURL string) (RuntimeStats, error)
}
StatsSource reads a load snapshot from a runtime serving at a loopback base URL. There is one implementation per runtime dialect, because the metric names differ. It returns a snapshot with Known false (and a nil error) when the runtime is reachable but reports nothing usable, and an error only when the read itself fails, which the manager turns into an unknown reading.
Source Files