Documentation
¶
Overview ¶
Package slot enforces modeld's single active local model invariant.
Index ¶
- type Option
- type Service
- func (s *Service) Describe(ctx context.Context, req transport.OpenSessionRequest) (transport.ModelInfo, error)
- func (s *Service) Embed(ctx context.Context, req transport.EmbedRequest) (transport.EmbedResult, error)
- func (s *Service) LoadModel(ctx context.Context, req transport.LoadModelRequest) (transport.ActiveModel, error)
- func (s *Service) OpenSession(ctx context.Context, req transport.OpenSessionRequest) (transport.Session, error)
- func (s *Service) StartReaper(ctx context.Context)
- func (s *Service) Status(ctx context.Context) (transport.DaemonStatus, error)
- func (s *Service) UnloadModel(ctx context.Context, req transport.UnloadModelRequest) error
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Option ¶
type Option func(*Service)
Option configures a slot Service.
func WithBackend ¶
WithBackend records the backend mode for Status.
func WithDataRoot ¶ added in v0.33.0
func WithIdleTTL ¶ added in v0.32.6
WithIdleTTL sets how long a resident model may sit idle before the background reaper releases it (freeing device memory so the GPU can return to idle). Zero (the default) disables idle reaping. Reaping requires StartReaper.
type Service ¶
type Service struct {
// contains filtered or unexported fields
}
Service wraps a backend transport.Service and exposes it as one active model slot. The wrapped backend still owns native inference; this layer owns residency cardinality, generation fencing, and switch/unload safety.
func (*Service) Describe ¶
func (s *Service) Describe(ctx context.Context, req transport.OpenSessionRequest) (transport.ModelInfo, error)
Describe stays side-effect-free: it never evicts the resident session.
- Same identity resident: the truthful answer is what that session actually resolved at open — a live recompute would be encumbered by the session's own footprint and report a uselessly small hypothetical (the "panel shows 439 while the session serves 2891" bug).
- Different identity resident: pass the resident session's footprint as ReclaimableBytes so the backend's free-memory snapshot reflects what an actual OpenSession would see after the switch closed the old session.
func (*Service) Embed ¶
func (s *Service) Embed(ctx context.Context, req transport.EmbedRequest) (transport.EmbedResult, error)
func (*Service) LoadModel ¶
func (s *Service) LoadModel(ctx context.Context, req transport.LoadModelRequest) (transport.ActiveModel, error)
func (*Service) OpenSession ¶
func (*Service) StartReaper ¶ added in v0.32.6
StartReaper runs the idle-reaper loop until ctx is canceled. While modeld holds the lease it periodically releases a resident model that has sat idle past the configured TTL, so device memory is freed and the GPU can return to idle (the 24/7-on-a-laptop case). It is a no-op when the TTL is zero. The tick is CPU-only and never touches the GPU.
Source Files