Affected by 
Documentation
¶
Index ¶
- func Execute(args []string) error
- type Client
- func (c *Client) ApplyChatTemplate(ctx context.Context, req llm.ChatRequest) (string, error)
- func (c *Client) Chat(ctx context.Context, req llm.ChatRequest, fn func(llm.ChatResponse)) error
- func (c *Client) Close() error
- func (c *Client) Completion(ctx context.Context, req llm.CompletionRequest, ...) error
- func (c *Client) ContextLength() int
- func (c *Client) Detokenize(ctx context.Context, tokens []int) (string, error)
- func (c *Client) Embedding(ctx context.Context, input string) ([]float32, int, error)
- func (c *Client) GetDeviceInfos(ctx context.Context) []ml.DeviceInfo
- func (c *Client) GetPort() int
- func (c *Client) HasExited() bool
- func (c *Client) Load(ctx context.Context, _ ml.SystemInfo, gpus []ml.DeviceInfo, requireFull bool) ([]ml.DeviceID, error)
- func (c *Client) MemorySize() (total, vram uint64)
- func (c *Client) ModelPath() string
- func (c *Client) Pid() int
- func (c *Client) Ping(ctx context.Context) error
- func (c *Client) Tokenize(ctx context.Context, content string) ([]int, error)
- func (c *Client) VRAMByGPU(id ml.DeviceID) uint64
- func (c *Client) WaitUntilRunning(ctx context.Context) error
- type CompletionRequest
- type CompletionResponse
- type Request
- type Runner
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Client ¶ added in v0.16.0
type Client struct {
// contains filtered or unexported fields
}
Client wraps an MLX runner subprocess to implement llm.LlamaServer for LLM models.
func NewClient ¶ added in v0.16.2
NewClient prepares a new MLX runner client for LLM models. The subprocess is not started until Load() is called.
func (*Client) ApplyChatTemplate ¶ added in v0.30.0
func (*Client) Chat ¶ added in v0.30.0
func (c *Client) Chat(ctx context.Context, req llm.ChatRequest, fn func(llm.ChatResponse)) error
func (*Client) Completion ¶ added in v0.16.0
func (c *Client) Completion(ctx context.Context, req llm.CompletionRequest, fn func(llm.CompletionResponse)) error
Completion implements llm.LlamaServer.
func (*Client) ContextLength ¶ added in v0.16.0
func (*Client) Detokenize ¶ added in v0.16.0
Detokenize implements llm.LlamaServer.
func (*Client) GetDeviceInfos ¶ added in v0.16.0
func (c *Client) GetDeviceInfos(ctx context.Context) []ml.DeviceInfo
GetDeviceInfos implements llm.LlamaServer.
func (*Client) Load ¶ added in v0.16.0
func (c *Client) Load(ctx context.Context, _ ml.SystemInfo, gpus []ml.DeviceInfo, requireFull bool) ([]ml.DeviceID, error)
Load checks whether the model fits in GPU memory and starts the subprocess.
func (*Client) MemorySize ¶ added in v0.17.5
MemorySize implements llm.LlamaServer.
type CompletionRequest ¶ added in v0.21.1
type CompletionResponse ¶ added in v0.17.5
type Request ¶
type Request struct {
CompletionRequest
Responses chan CompletionResponse
Pipeline func(context.Context, Request) error
Ctx context.Context //nolint:containedctx // Queued requests carry caller cancellation to the runner.
Tokens []int32
SamplerOpts sample.Options
}
Request is a short-lived struct that carries a completion request through a channel from the HTTP handler to the runner goroutine. The ctx field must travel with the request so that cancellation propagates across the channel boundary.
type Runner ¶ added in v0.18.0
type Runner struct {
Model base.Model
Tokenizer *tokenizer.Tokenizer
Requests chan Request
Sampler *sample.Sampler
// contains filtered or unexported fields
}
func (*Runner) Prepare ¶ added in v0.21.1
Prepare tokenizes the prompt and validates it against the model's context length. It is safe to call from any goroutine. On success it populates request.Tokens and adjusts request.Options.NumPredict.
Source Files
Directories