vllm

package
v0.2.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 15, 2026 License: MIT Imports: 8 Imported by: 0

Documentation

Overview

Package vllm provides vLLM OpenAI-compatible API integration for the LLM gateway.

Package vllm provides vLLM OpenAI-compatible API integration for the LLM gateway.

Index

Constants

This section is empty.

Variables

View Source
var Registration = providers.Registration{
	Type:                        "vllm",
	New:                         New,
	PassthroughSemanticEnricher: passthroughSemanticEnricher,
	Discovery: providers.DiscoveryConfig{
		DefaultBaseURL:  defaultBaseURL,
		AllowAPIKeyless: true,
	},
}

Registration provides factory registration for the vLLM provider.

Functions

func New

New creates a new vLLM provider.

Types

type Provider

type Provider struct {
	// contains filtered or unexported fields
}

Provider implements the core.Provider interface for vLLM.

func (*Provider) ChatCompletion

func (p *Provider) ChatCompletion(ctx context.Context, req *core.ChatRequest) (*core.ChatResponse, error)

ChatCompletion sends a chat completion request to vLLM.

func (*Provider) Embeddings

Embeddings sends an embeddings request to vLLM.

func (*Provider) ListModels

func (p *Provider) ListModels(ctx context.Context) (*core.ModelsResponse, error)

ListModels retrieves the list of available models from vLLM. ListModels lists the served models, keeping the max_model_len vLLM reports per entry as the context window.

func (*Provider) Passthrough

Passthrough routes an opaque provider-native request to vLLM.

func (*Provider) Responses

Responses sends a Responses API request to vLLM.

func (*Provider) SetBaseURL

func (p *Provider) SetBaseURL(url string)

SetBaseURL allows configuring a custom base URL for the provider.

func (*Provider) StreamChatCompletion

func (p *Provider) StreamChatCompletion(ctx context.Context, req *core.ChatRequest) (io.ReadCloser, error)

StreamChatCompletion returns a raw response body for streaming.

func (*Provider) StreamResponses

func (p *Provider) StreamResponses(ctx context.Context, req *core.ResponsesRequest) (io.ReadCloser, error)

StreamResponses streams a Responses API request to vLLM.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL