llamacpp

package
v0.2.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 15, 2026 License: MIT Imports: 9 Imported by: 0

Documentation

Overview

Package llamacpp provides llama.cpp llama-server OpenAI-compatible API integration for the LLM gateway. The same provider type fits LM Studio and other plain OpenAI-compatible local servers.

Index

Constants

This section is empty.

Variables

View Source
var Registration = providers.Registration{
	Type:                        "llamacpp",
	New:                         New,
	PassthroughSemanticEnricher: passthroughSemanticEnricher,
	Discovery: providers.DiscoveryConfig{
		RequireBaseURL:  true,
		AllowAPIKeyless: true,
	},
}

Registration provides factory registration for the llama.cpp provider. llama-server's default port (8080) collides with the gateway's own, so the base URL is required rather than defaulted; the API key is optional and only needed when llama-server was started with --api-key.

Functions

func New

New creates a new llama.cpp provider.

Types

type Provider

type Provider struct {
	// contains filtered or unexported fields
}

Provider implements the core.Provider interface for llama.cpp.

func (*Provider) ChatCompletion

func (p *Provider) ChatCompletion(ctx context.Context, req *core.ChatRequest) (*core.ChatResponse, error)

ChatCompletion sends a chat completion request to llama-server.

func (*Provider) Embeddings

Embeddings sends an embeddings request to llama-server. The loaded model must use a pooling type other than none for llama-server to serve it.

func (*Provider) ListModels

func (p *Provider) ListModels(ctx context.Context) (*core.ModelsResponse, error)

ListModels retrieves the list of available models from llama-server, keeping the context window and modalities the server reports for the model it has loaded so clients can size requests without a model-registry entry.

func (*Provider) Passthrough

Passthrough routes an opaque provider-native request to llama-server. OpenAI-shaped endpoints go through the /v1 base; llama-server's native endpoints (/health, /props, /slots, /rerank, /infill, /tokenize, ...) live at the server root.

func (*Provider) Responses

Responses sends a Responses API request to llama-server.

func (*Provider) SetBaseURL

func (p *Provider) SetBaseURL(url string)

SetBaseURL allows configuring a custom base URL for the provider.

func (*Provider) StreamChatCompletion

func (p *Provider) StreamChatCompletion(ctx context.Context, req *core.ChatRequest) (io.ReadCloser, error)

StreamChatCompletion returns a raw response body for streaming.

func (*Provider) StreamResponses

func (p *Provider) StreamResponses(ctx context.Context, req *core.ResponsesRequest) (io.ReadCloser, error)

StreamResponses streams a Responses API request to llama-server.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL