Documentation
¶
Overview ¶
Package vllm provides a spec-faithful HTTP emulator of the vLLM OpenAI-compatible API server for use in tests.
Index ¶
Constants ¶
View Source
const ChatStreamWithUsageSSE = "data: {\"id\":\"gen-vllm-refbackend-stream\",\"object\":\"chat.completion.chunk\",\"created\":1715620000,\"model\":\"meta-llama/Llama-3-8B-Instruct\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"vllm-ok\"},\"finish_reason\":null}]}\n\n" +
"data: {\"id\":\"gen-vllm-refbackend-stream\",\"object\":\"chat.completion.chunk\",\"created\":1715620000,\"model\":\"meta-llama/Llama-3-8B-Instruct\",\"choices\":[{\"index\":0,\"delta\":{},\"finish_reason\":\"stop\"}]}\n\n" +
"data: {\"id\":\"gen-vllm-refbackend-stream\",\"object\":\"chat.completion.chunk\",\"created\":1715620000,\"model\":\"meta-llama/Llama-3-8B-Instruct\",\"choices\":[],\"usage\":{\"prompt_tokens\":3,\"completion_tokens\":7,\"total_tokens\":10}}\n\n" +
"data: [DONE]\n\n"
ChatStreamWithUsageSSE includes a final usage chunk before [DONE].
View Source
const DefaultChatNonStreamJSON = `` /* 310-byte string literal not displayed */
DefaultChatNonStreamJSON is a minimal Chat Completions response for vLLM.
View Source
const DefaultChatStreamSSE = "data: {\"id\":\"gen-vllm-refbackend-stream\",\"object\":\"chat.completion.chunk\",\"created\":1715620000,\"model\":\"meta-llama/Llama-3-8B-Instruct\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"vllm-ok\"},\"finish_reason\":null}]}\n\n" +
"data: [DONE]\n\n"
DefaultChatStreamSSE is a minimal streaming Chat Completions response.
View Source
const DefaultModelsJSON = `` /* 176-byte string literal not displayed */
DefaultModelsJSON is a minimal models list response.
View Source
const DefaultResponsesNonStreamJSON = `` /* 438-byte string literal not displayed */
DefaultResponsesNonStreamJSON is a minimal Responses API completed response.
View Source
const DefaultResponsesStreamSSE = "event: response.completed\n" +
"data: {\"type\":\"response.completed\",\"sequence_number\":1,\"response\":{\"id\":\"resp_vllm_refbackend_stream\",\"object\":\"response\",\"created_at\":1715620000,\"status\":\"completed\",\"model\":\"meta-llama/Llama-3-8B-Instruct\",\"output\":[{\"type\":\"message\",\"id\":\"m1\",\"status\":\"completed\",\"role\":\"assistant\",\"content\":[{\"type\":\"output_text\",\"text\":\"vllm-ok\"}]}],\"usage\":{\"input_tokens\":3,\"output_tokens\":7,\"total_tokens\":10}}}\n\n" +
"data: [DONE]\n\n"
DefaultResponsesStreamSSE is a minimal Responses SSE stream.
Variables ¶
This section is empty.
Functions ¶
func NewHandler ¶
NewHandler returns an http.Handler that emulates the vLLM OpenAI-compatible API server at http://localhost:8000/v1.
Types ¶
type Config ¶
type Config struct {
AllowMissingBearer bool
OnAuthorizedCredential func(secret string)
ForcedHTTPStatus int
ForcedRetryAfter string
ForcedErrorJSON string
OnRequestBody func(body []byte)
OnRequestHeaders func(h http.Header)
ChatNonStreamJSON string
ChatStreamSSE string
ResponsesNonStreamJSON string
ResponsesStreamSSE string
}
Config tunes the emulator handler.
Source Files
Click to show internal directories.
Click to hide internal directories.