Documentation
¶
Overview ¶
Package grpc is the gRPC wire transport for the runtime/transport.Service contract. It lives in the runtime (which owns the contract) and is imported by modeld to serve it, so the dependency arrow stays one-way: modeld -> runtime. Both the server (serves any transport.Service) and the client (dials the lease leader and implements transport.Service/Session) live here.
The bindings are hand-written against a registered JSON codec rather than generated from a .proto, so the transport is live without a protoc toolchain. Payloads are plain Go structs (the transport.* types are JSON-friendly).
Index ¶
- func Serve(ctx context.Context, lis net.Listener, svc transport.Service, ...) error
- type Client
- func (c *Client) Close() error
- func (c *Client) Describe(ctx context.Context, req transport.OpenSessionRequest) (transport.ModelInfo, error)
- func (c *Client) DiskStats(ctx context.Context) (transport.NodeDiskStats, error)
- func (c *Client) Embed(ctx context.Context, req transport.EmbedRequest) (transport.EmbedResult, error)
- func (c *Client) Health(ctx context.Context) (HealthStatus, error)
- func (c *Client) ListModels(ctx context.Context) ([]transport.NodeModel, error)
- func (c *Client) LoadModel(ctx context.Context, req transport.LoadModelRequest) (transport.ActiveModel, error)
- func (c *Client) OpenSession(ctx context.Context, req transport.OpenSessionRequest) (transport.Session, error)
- func (c *Client) PushModel(ctx context.Context, manifest transport.PushManifest, r io.Reader) (transport.PushResult, error)
- func (c *Client) RemoveModel(ctx context.Context, name string) error
- func (c *Client) Status(ctx context.Context) (transport.DaemonStatus, error)
- func (c *Client) UnloadModel(ctx context.Context, req transport.UnloadModelRequest) error
- type HealthStatus
- type Server
- func (s *Server) HandleConn(ctx context.Context, st stats.ConnStats)
- func (s *Server) HandleRPC(context.Context, stats.RPCStats)
- func (s *Server) Register(reg grpclib.ServiceRegistrar)
- func (s *Server) TagConn(ctx context.Context, _ *stats.ConnTagInfo) context.Context
- func (s *Server) TagRPC(ctx context.Context, _ *stats.RPCTagInfo) context.Context
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client dials a modeld owner endpoint and implements transport.Service over the wire. owner is the lease instance id it expects to be serving; it is fenced on every call so a client never acts on a stale owner after a takeover.
func DialLeader ¶
func DialLeader(endpoint, expectedOwner string, opts ...grpclib.DialOption) (*Client, error)
DialLeader connects to the owner's advertised endpoint and fences every call with the expected owner instance id resolved from the lease record.
func (*Client) Describe ¶
func (c *Client) Describe(ctx context.Context, req transport.OpenSessionRequest) (transport.ModelInfo, error)
Describe reports a model's capabilities as read by the daemon from the model metadata and capacity policy. The model is identified by req's typed handle (Type + Path); Config carries the requested context/runtime knobs for the capacity plan.
func (*Client) DiskStats ¶ added in v0.34.0
DiskStats reports free/used/total bytes on the filesystem backing the node's models directory.
func (*Client) Embed ¶ added in v0.32.4
func (c *Client) Embed(ctx context.Context, req transport.EmbedRequest) (transport.EmbedResult, error)
Embed computes a one-shot embedding through the owner without opening a persistent decode session.
func (*Client) Health ¶
func (c *Client) Health(ctx context.Context) (HealthStatus, error)
Health pings the owner for liveness. It is not fenced — the caller uses the returned InstanceID to confirm the lease holder is the process answering.
func (*Client) ListModels ¶ added in v0.34.0
ListModels enumerates every model on the node's models directory.
func (*Client) LoadModel ¶ added in v0.32.4
func (c *Client) LoadModel(ctx context.Context, req transport.LoadModelRequest) (transport.ActiveModel, error)
func (*Client) OpenSession ¶
func (*Client) PushModel ¶ added in v0.34.0
func (c *Client) PushModel(ctx context.Context, manifest transport.PushManifest, r io.Reader) (transport.PushResult, error)
PushModel streams a model's bytes to the node from r, which manifest describes. The node verifies the stream against manifest.Digest as it arrives and installs the model atomically; see transport.NodeAdmin.
func (*Client) RemoveModel ¶ added in v0.34.0
RemoveModel deletes a model from the node's models directory.
func (*Client) UnloadModel ¶ added in v0.32.4
type HealthStatus ¶
type HealthStatus struct {
InstanceID string // the owner instance actually serving this endpoint
Ready bool
Backend string // inference backend the owner serves ("llama"/"openvino"/"none")
}
HealthStatus is the result of the unfenced liveness probe.
type Server ¶
type Server struct {
// contains filtered or unexported fields
}
Server adapts a transport.Service to the gRPC wire. It is generic: modeld supplies the concrete (CGO-backed) Service and the owner instance id from the lease. Sessions live here keyed by an opaque handle that encodes the owner instance, so a handle minted by a previous owner is rejected after takeover.
func NewServer ¶
NewServer wraps a transport.Service. instanceID is the owner's lease instance id used for fencing; pass "" to disable fencing (the unwired/local path). backend is the served inference backend ("llama"/"openvino"/"none"/"") echoed on the health probe so the runtime learns the daemon's mode over the wire.
func (*Server) HandleConn ¶ added in v0.32.6
func (*Server) Register ¶
func (s *Server) Register(reg grpclib.ServiceRegistrar)
Register mounts the service on a gRPC server.
Source Files