Documentation
¶
Overview ¶
Package slot enforces modeld's single active local model invariant.
Index ¶
- type Option
- type Service
- func (s *Service) Describe(ctx context.Context, req transport.OpenSessionRequest) (transport.ModelInfo, error)
- func (s *Service) Embed(ctx context.Context, req transport.EmbedRequest) (transport.EmbedResult, error)
- func (s *Service) LoadModel(ctx context.Context, req transport.LoadModelRequest) (transport.ActiveModel, error)
- func (s *Service) OpenSession(ctx context.Context, req transport.OpenSessionRequest) (transport.Session, error)
- func (s *Service) StartReaper(ctx context.Context)
- func (s *Service) Status(ctx context.Context) (transport.DaemonStatus, error)
- func (s *Service) UnloadModel(ctx context.Context, req transport.UnloadModelRequest) error
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Option ¶
type Option func(*Service)
Option configures a slot Service.
func WithBackend ¶
WithBackend records the backend mode for Status.
func WithIdleTTL ¶ added in v0.32.6
WithIdleTTL sets how long a resident model may sit idle before the background reaper releases it (freeing device memory so the GPU can return to idle). Zero (the default) disables idle reaping. Reaping requires StartReaper.
type Service ¶
type Service struct {
// contains filtered or unexported fields
}
Service wraps a backend transport.Service and exposes it as one active model slot. The wrapped backend still owns native inference; this layer owns residency cardinality, generation fencing, and switch/unload safety.
func (*Service) Embed ¶
func (s *Service) Embed(ctx context.Context, req transport.EmbedRequest) (transport.EmbedResult, error)
func (*Service) LoadModel ¶
func (s *Service) LoadModel(ctx context.Context, req transport.LoadModelRequest) (transport.ActiveModel, error)
func (*Service) OpenSession ¶
func (*Service) StartReaper ¶ added in v0.32.6
StartReaper runs the idle-reaper loop until ctx is canceled. While modeld holds the lease it periodically releases a resident model that has sat idle past the configured TTL, so device memory is freed and the GPU can return to idle (the 24/7-on-a-laptop case). It is a no-op when the TTL is zero. The tick is CPU-only and never touches the GPU.
Source Files