Documentation
¶
Overview ¶
Package modeldconn is the runtime's client seam to the modeld daemon: it resolves the current lease leader (via modeldprobe), dials it over the gRPC transport, and opens sessions. Local backend providers (llama, openvino) call OpenSession instead of constructing an in-process CGO session, so the runtime stays pure Go and talks to modeld only through runtime/transport.
Index ¶
- Constants
- func Available() bool
- func Backend() string
- func CloseEndpoint(backendID string)
- func Describe(ctx context.Context, ref ModelRef, cfg transport.Config) (transport.ModelInfo, error)
- func DescribeTarget(ctx context.Context, target ModeldTarget, ref ModelRef, cfg transport.Config) (transport.ModelInfo, error)
- func Embed(ctx context.Context, ref ModelRef, cfg transport.Config, text string) (transport.EmbedResult, error)
- func EmbedTarget(ctx context.Context, target ModeldTarget, ref ModelRef, cfg transport.Config, ...) (transport.EmbedResult, error)
- func ListModelsTarget(ctx context.Context, target ModeldTarget) ([]transport.NodeModel, error)
- func LoadModel(ctx context.Context, ref ModelRef, cfg transport.Config, ...) (transport.ActiveModel, error)
- func LocalEndpointAddr(ctx context.Context) (string, error)
- func OpenSession(ctx context.Context, ref ModelRef, cfg transport.Config) (transport.Session, error)
- func OpenSessionTarget(ctx context.Context, target ModeldTarget, ref ModelRef, cfg transport.Config) (transport.Session, error)
- func PushModelTarget(ctx context.Context, target ModeldTarget, manifest transport.PushManifest, ...) (transport.PushResult, error)
- func ServeableBackend() string
- func SetDataRoot(root string)
- func SnapshotDir(backend string) string
- func Status(ctx context.Context) (transport.DaemonStatus, error)
- func UnloadModel(ctx context.Context, expectedGeneration uint64) error
- type EndpointClient
- type EndpointHealth
- type ModelRef
- type ModeldTarget
Constants ¶
const LocalSentinel = "local"
LocalSentinel is the backend BaseURL value meaning "this modeld backend row is the local daemon" — reach it via the lease (LocalEndpointAddr), not a stored address. Mirrors the existing "local" compatibility alias already accepted for --type.
Variables ¶
This section is empty.
Functions ¶
func Available ¶
func Available() bool
Available is the cheap, offline check (lease inspection, no network): is a modeld owner currently holding a fresh lease? Providers use it to gate capability advertisement without a round-trip per call.
func Backend ¶
func Backend() string
Backend is the cheap, offline check for the inference backend the running modeld owner serves ("llama"/"openvino"/"none"), read from the lease. Empty when no owner holds a fresh lease. Providers gate on this so the backend modeld is NOT in advertises no local capability instead of failing deep in the engine.
func CloseEndpoint ¶ added in v0.34.0
func CloseEndpoint(backendID string)
CloseEndpoint drops and closes a cached endpoint connection, e.g. when its backend row is deleted from the registry.
func Describe ¶
Describe asks the running modeld owner for a model's capabilities, read from the model metadata by the backend that serves it. This is the modeld→runtime info-flow for model facts (e.g. the trained context window): the runtime is the consumer and never parses model files itself. A not-running/unreachable owner surfaces the probe's typed error.
func DescribeTarget ¶ added in v0.34.0
func DescribeTarget(ctx context.Context, target ModeldTarget, ref ModelRef, cfg transport.Config) (transport.ModelInfo, error)
DescribeTarget is the targeted equivalent of Describe.
func Embed ¶ added in v0.32.4
func Embed(ctx context.Context, ref ModelRef, cfg transport.Config, text string) (transport.EmbedResult, error)
Embed asks the running modeld owner to compute a one-shot embedding for text. It uses the same typed model handle and owner fence as Describe, but does not open a persistent session.
func EmbedTarget ¶ added in v0.34.0
func EmbedTarget(ctx context.Context, target ModeldTarget, ref ModelRef, cfg transport.Config, text string) (transport.EmbedResult, error)
EmbedTarget is the targeted equivalent of Embed.
func ListModelsTarget ¶ added in v0.34.0
ListModelsTarget, PushModelTarget etc. for the ModeldTarget wrapper.
func LoadModel ¶ added in v0.32.4
func LoadModel(ctx context.Context, ref ModelRef, cfg transport.Config, expectedGeneration ...uint64) (transport.ActiveModel, error)
LoadModel explicitly activates the single modeld slot, switching away from an idle active model when needed. expectedGeneration, when provided, makes the switch conditional on the caller's current slot view.
func LocalEndpointAddr ¶ added in v0.34.0
LocalEndpointAddr resolves the local modeld owner's current advertised endpoint address via the lease, for callers that want to reach it through the same fenced Endpoint path used for remote nodes (observe-only backend reconcile) rather than the hot chat-serving path's dedicated single-slot cache (OpenSession/Describe/Embed/LoadModel above, untouched by this file).
func OpenSession ¶
func OpenSession(ctx context.Context, ref ModelRef, cfg transport.Config) (transport.Session, error)
OpenSession resolves the modeld leader, confirms it actually answers a health probe, and opens a session on it. The returned session is resident in modeld; the caller drives EnsurePrefix/PrefillSuffix/Decode over the wire. A not-running/unreachable owner surfaces the probe's typed error; a model typed for a backend the daemon does not serve surfaces transport.ErrBackendMismatch.
func OpenSessionTarget ¶ added in v0.34.0
func OpenSessionTarget(ctx context.Context, target ModeldTarget, ref ModelRef, cfg transport.Config) (transport.Session, error)
OpenSessionTarget opens using the target. If target.Endpoint == "", falls back to the lease-based local path (preserves all existing behavior for implicit "local" modeld usage).
func PushModelTarget ¶ added in v0.34.0
func PushModelTarget(ctx context.Context, target ModeldTarget, manifest transport.PushManifest, r io.Reader) (transport.PushResult, error)
func ServeableBackend ¶ added in v0.32.6
func ServeableBackend() string
ServeableBackend reports the inference backend modeld can serve, smoothed over brief lease gaps so local-model capability does not flap during a daemon restart. While a fresh lease is held it returns the live backend and refreshes the cache; during a gap it returns the last-observed backend for up to serveableGraceWindow; after that it returns "". Capability advertisement (provider SessionAvailable / catalog listing) uses this; live-decode paths keep using the strict Backend()/Available() so they fail honestly when modeld is actually gone.
func SetDataRoot ¶
func SetDataRoot(root string)
SetDataRoot overrides the data root used to locate the owner lease. It also invalidates the serveable-backend grace cache: a different data root is a different modeld context, so a backend observed under the old root must not keep being advertised under the new one.
func SnapshotDir ¶ added in v0.33.0
SnapshotDir resolves the on-disk directory a backend's warm cache should persist durable session snapshots to: $CONTENOX_WARM_SNAPSHOT_DIR/<backend> when set, else <data root>/modeld-snapshots/<backend>. It returns "" when $CONTENOX_WARM_SNAPSHOT_DISABLE is set, which the caller (a modelrepo.DiskSnapshotStore) treats as "no snapshot dir" — Save/Load become no-ops and the warm cache behaves exactly as it did before snapshot wiring existed.
Types ¶
type EndpointClient ¶ added in v0.34.0
type EndpointClient struct {
*transportgrpc.Client
EndpointHealth
}
EndpointClient is a fenced connection to one modeld node (local or remote), reached by address rather than lease discovery.
func Endpoint ¶ added in v0.34.0
func Endpoint(ctx context.Context, backendID, addr string) (EndpointClient, error)
Endpoint dials (or reuses a cached connection to) the modeld node identified by backendID at addr. It health-probes first — unfenced, since the caller has no lease file to read a remote node's owner instance id from — to learn the node's current instance id, then dials (or confirms) a client fenced to it. A cached connection whose remembered instance id no longer matches the node's live Health response (the node restarted) is dropped and redialed automatically, mirroring the local dial()'s take-over-redials behavior above.
type EndpointHealth ¶ added in v0.34.0
type EndpointHealth struct {
InstanceID string
Backend string // "llama" | "openvino" | "none"
Ready bool
}
EndpointHealth reports what a node's Health RPC returned, cached alongside its connection.
type ModelRef ¶
type ModelRef struct {
Name string
Type string
Digest string
Path string
// Adapters are LoRA adapters applied to this model variant, in order. Empty =
// the base model. They are part of the cache identity: base+A and base+B must
// not share a resident session.
Adapters []transport.AdapterSpec
}
ModelRef is the typed model handle the runtime passes to modeld: a logical Name + backend Type + content Digest form the cache identity, and Path is the runtime-resolved on-disk location (GGUF file or IR directory) the daemon loads from. Type lets the daemon reject a model it does not serve.
type ModeldTarget ¶ added in v0.34.0
type ModeldTarget struct {
BackendID string
Endpoint string // host:port or empty for lease-resolved local
Instance string // owner instance for fencing (empty when using lease path)
}
ModeldTarget describes a specific modeld to talk to (local lease or remote). BackendID is the registered backend row ID (for caching / logging). When Endpoint == "" the target is resolved via the local lease (default local modeld).
Source Files