Documentation
¶
Overview ¶
Package llamacpp provides llama.cpp llama-server OpenAI-compatible API integration for the LLM gateway. The same provider type fits LM Studio and other plain OpenAI-compatible local servers.
Index ¶
- Variables
- func New(cfg providers.ProviderConfig, opts providers.ProviderOptions) core.Provider
- type Provider
- func (p *Provider) ChatCompletion(ctx context.Context, req *core.ChatRequest) (*core.ChatResponse, error)
- func (p *Provider) Embeddings(ctx context.Context, req *core.EmbeddingRequest) (*core.EmbeddingResponse, error)
- func (p *Provider) ListModels(ctx context.Context) (*core.ModelsResponse, error)
- func (p *Provider) Passthrough(ctx context.Context, req *core.PassthroughRequest) (*core.PassthroughResponse, error)
- func (p *Provider) Responses(ctx context.Context, req *core.ResponsesRequest) (*core.ResponsesResponse, error)
- func (p *Provider) SetBaseURL(url string)
- func (p *Provider) StreamChatCompletion(ctx context.Context, req *core.ChatRequest) (io.ReadCloser, error)
- func (p *Provider) StreamResponses(ctx context.Context, req *core.ResponsesRequest) (io.ReadCloser, error)
Constants ¶
This section is empty.
Variables ¶
var Registration = providers.Registration{ Type: "llamacpp", New: New, PassthroughSemanticEnricher: passthroughSemanticEnricher, Discovery: providers.DiscoveryConfig{ RequireBaseURL: true, AllowAPIKeyless: true, }, }
Registration provides factory registration for the llama.cpp provider. llama-server's default port (8080) collides with the gateway's own, so the base URL is required rather than defaulted; the API key is optional and only needed when llama-server was started with --api-key.
Functions ¶
func New ¶
func New(cfg providers.ProviderConfig, opts providers.ProviderOptions) core.Provider
New creates a new llama.cpp provider.
Types ¶
type Provider ¶
type Provider struct {
// contains filtered or unexported fields
}
Provider implements the core.Provider interface for llama.cpp.
func (*Provider) ChatCompletion ¶
func (p *Provider) ChatCompletion(ctx context.Context, req *core.ChatRequest) (*core.ChatResponse, error)
ChatCompletion sends a chat completion request to llama-server.
func (*Provider) Embeddings ¶
func (p *Provider) Embeddings(ctx context.Context, req *core.EmbeddingRequest) (*core.EmbeddingResponse, error)
Embeddings sends an embeddings request to llama-server. The loaded model must use a pooling type other than none for llama-server to serve it.
func (*Provider) ListModels ¶
ListModels retrieves the list of available models from llama-server, keeping the context window and modalities the server reports for the model it has loaded so clients can size requests without a model-registry entry.
func (*Provider) Passthrough ¶
func (p *Provider) Passthrough(ctx context.Context, req *core.PassthroughRequest) (*core.PassthroughResponse, error)
Passthrough routes an opaque provider-native request to llama-server. OpenAI-shaped endpoints go through the /v1 base; llama-server's native endpoints (/health, /props, /slots, /rerank, /infill, /tokenize, ...) live at the server root.
func (*Provider) Responses ¶
func (p *Provider) Responses(ctx context.Context, req *core.ResponsesRequest) (*core.ResponsesResponse, error)
Responses sends a Responses API request to llama-server.
func (*Provider) SetBaseURL ¶
SetBaseURL allows configuring a custom base URL for the provider.
func (*Provider) StreamChatCompletion ¶
func (p *Provider) StreamChatCompletion(ctx context.Context, req *core.ChatRequest) (io.ReadCloser, error)
StreamChatCompletion returns a raw response body for streaming.
func (*Provider) StreamResponses ¶
func (p *Provider) StreamResponses(ctx context.Context, req *core.ResponsesRequest) (io.ReadCloser, error)
StreamResponses streams a Responses API request to llama-server.