Documentation
¶
Overview ¶
Package openai_compatible provides a reusable implementation for LLM providers that use OpenAI-compatible APIs.
This includes providers like OpenAI, Groq, Together AI, Fireworks AI, and others that implement the OpenAI chat completions API format.
Basic usage:
provider, _ := openai_compatible.New("groq", "gsk_xxx", "https://api.groq.com")
proxy := llmproxy.NewProxy(provider)
Index ¶
- func ParseOpenAIRequest(body io.ReadCloser) (llmproxy.BodyMetadata, []byte, error)
- func ParseOpenAIRequestBody(data []byte) (llmproxy.BodyMetadata, error)
- func ParseResponsesRequest(body io.ReadCloser) (llmproxy.BodyMetadata, []byte, error)
- func ParseResponsesRequestBody(data []byte) (llmproxy.BodyMetadata, error)
- type CompletionTokensDetails
- type Enricher
- type Extractor
- type MultiAPIExtractor
- type MultiAPIParser
- type OpenAIRequest
- type OpenAIResponse
- type Parser
- type PromptTokensDetails
- type Provider
- type Resolver
- type ResponseChoice
- type ResponseMessage
- type ResponsesError
- type ResponsesExtractor
- type ResponsesInputDetails
- type ResponsesOutputAnnotation
- type ResponsesOutputContent
- type ResponsesOutputDetails
- type ResponsesOutputItem
- type ResponsesOutputSummary
- type ResponsesParser
- type ResponsesRequest
- type ResponsesResponse
- type ResponsesStreamingExtractor
- type ResponsesUsage
- type StreamingExtractor
- type StreamingMultiAPIExtractor
- type UsageInfo
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ParseOpenAIRequest ¶
func ParseOpenAIRequest(body io.ReadCloser) (llmproxy.BodyMetadata, []byte, error)
ParseOpenAIRequest is a convenience function that parses an OpenAI-compatible request body and returns the metadata and raw bytes.
func ParseOpenAIRequestBody ¶
func ParseOpenAIRequestBody(data []byte) (llmproxy.BodyMetadata, error)
ParseOpenAIRequestBody parses raw JSON bytes as an OpenAI-compatible request. It returns only the metadata, not the raw bytes.
func ParseResponsesRequest ¶ added in v0.0.6
func ParseResponsesRequest(body io.ReadCloser) (llmproxy.BodyMetadata, []byte, error)
func ParseResponsesRequestBody ¶ added in v0.0.6
func ParseResponsesRequestBody(data []byte) (llmproxy.BodyMetadata, error)
Types ¶
type CompletionTokensDetails ¶ added in v0.0.3
type CompletionTokensDetails struct {
ReasoningTokens int `json:"reasoning_tokens,omitempty"`
AudioTokens int `json:"audio_tokens,omitempty"`
AcceptedPredictionTokens int `json:"accepted_prediction_tokens,omitempty"`
RejectedPredictionTokens int `json:"rejected_prediction_tokens,omitempty"`
}
CompletionTokensDetails contains detailed completion token breakdown.
type Enricher ¶
type Enricher struct {
// APIKey is the API key for authentication.
APIKey string
}
Enricher implements llmproxy.RequestEnricher for OpenAI-compatible APIs. It sets the required Authorization header with a Bearer token.
func NewEnricher ¶
NewEnricher creates a new enricher with the given API key.
type Extractor ¶
type Extractor struct{}
Extractor implements llmproxy.ResponseExtractor for OpenAI-compatible responses. It parses the response JSON and extracts token usage, choices, and other metadata.
func NewExtractor ¶
func NewExtractor() *Extractor
NewExtractor creates a new OpenAI-compatible response extractor.
func (*Extractor) Extract ¶
Extract reads the response body and parses it as an OpenAI-compatible response. It extracts the ID, model, usage statistics, and completion choices.
Returns:
- metadata: Parsed response metadata
- rawBody: The original response body bytes (preserved for forwarding)
- error: Any parsing error
The raw body is returned so it can be re-attached to the response for the caller, preserving any custom/unsupported fields in the original JSON.
type MultiAPIExtractor ¶ added in v0.0.6
type MultiAPIExtractor struct {
// contains filtered or unexported fields
}
func NewMultiAPIExtractor ¶ added in v0.0.6
func NewMultiAPIExtractor() *MultiAPIExtractor
func (*MultiAPIExtractor) Extract ¶ added in v0.0.6
func (e *MultiAPIExtractor) Extract(resp *http.Response) (llmproxy.ResponseMetadata, []byte, error)
type MultiAPIParser ¶ added in v0.0.6
type MultiAPIParser struct {
// contains filtered or unexported fields
}
func NewMultiAPIParser ¶ added in v0.0.6
func NewMultiAPIParser() *MultiAPIParser
func (*MultiAPIParser) Parse ¶ added in v0.0.6
func (p *MultiAPIParser) Parse(body io.ReadCloser) (llmproxy.BodyMetadata, []byte, error)
type OpenAIRequest ¶
type OpenAIRequest struct {
// Model is the model identifier (e.g., "gpt-4", "llama-2-70b").
Model string `json:"model"`
// Messages is the conversation history.
Messages []llmproxy.Message `json:"messages"`
// MaxTokens limits the generation length.
MaxTokens int `json:"max_tokens,omitempty"`
// Stream enables streaming responses.
Stream bool `json:"stream"`
// Custom holds provider-specific parameters not in the standard schema.
Custom map[string]interface{} `json:"-"`
}
OpenAIRequest represents an OpenAI-compatible chat completion request. It includes standard fields and captures custom fields for provider extensions.
func (*OpenAIRequest) UnmarshalJSON ¶
func (r *OpenAIRequest) UnmarshalJSON(data []byte) error
UnmarshalJSON implements custom JSON unmarshaling to capture unknown fields.
type OpenAIResponse ¶
type OpenAIResponse struct {
// ID is the unique response identifier.
ID string `json:"id"`
// Object is the object type (e.g., "chat.completion").
Object string `json:"object"`
// Created is the Unix timestamp of creation.
Created int64 `json:"created"`
// Model is the model used for completion.
Model string `json:"model"`
// Usage contains token consumption statistics.
Usage UsageInfo `json:"usage"`
// Choices contains the completion choices.
Choices []ResponseChoice `json:"choices"`
}
OpenAIResponse represents an OpenAI-compatible chat completion response.
type Parser ¶
type Parser struct{}
Parser implements llmproxy.BodyParser for OpenAI-compatible request formats. It extracts model, messages, and other fields into a unified BodyMetadata structure.
func (*Parser) Parse ¶
func (p *Parser) Parse(body io.ReadCloser) (llmproxy.BodyMetadata, []byte, error)
Parse reads an OpenAI-compatible request body and extracts metadata. It returns both the parsed metadata and the raw body bytes for later use.
The parser handles standard OpenAI fields and captures unknown fields in the Custom map for provider-specific extensions.
type PromptTokensDetails ¶ added in v0.0.3
type PromptTokensDetails struct {
CachedTokens int `json:"cached_tokens,omitempty"`
AudioTokens int `json:"audio_tokens,omitempty"`
}
PromptTokensDetails contains detailed prompt token breakdown.
type Provider ¶
type Provider struct {
*llmproxy.BaseProvider
}
Provider is an OpenAI-compatible provider implementation. It embeds llmproxy.BaseProvider and can be further customized.
func New ¶
New creates a new OpenAI-compatible provider with the given configuration.
Parameters:
- name: A unique identifier for the provider (e.g., "openai", "groq")
- apiKey: The API key for authentication
- baseURL: The provider's API base URL (e.g., "https://api.openai.com")
Example:
provider, _ := openai_compatible.New("groq", "gsk_xxx", "https://api.groq.com")
func NewMultiAPI ¶ added in v0.0.6
func NewWithProvider ¶
func NewWithProvider(name string, p *llmproxy.BaseProvider) *Provider
NewWithProvider creates a Provider that wraps an existing BaseProvider. Use this when you need to customize individual components before creating the provider.
Example:
base := llmproxy.NewBaseProvider("custom",
llmproxy.WithBodyParser(&Parser{}),
llmproxy.WithRequestEnricher(customEnricher),
)
provider := openai_compatible.NewWithProvider("custom", base)
func (*Provider) WebSocketURL ¶ added in v0.0.8
type Resolver ¶
func NewResolver ¶
func NewResolverWithAPIType ¶ added in v0.0.6
func (*Resolver) WebSocketURL ¶ added in v0.0.8
WebSocketURL converts the provider HTTP base URL to a WebSocket URL.
https://api.openai.com -> wss://api.openai.com/v1/responses http://localhost:8080 -> ws://localhost:8080/v1/responses
type ResponseChoice ¶
type ResponseChoice struct {
// Index is the choice position.
Index int `json:"index"`
// Message contains the completed message (non-streaming).
Message *ResponseMessage `json:"message,omitempty"`
// Delta contains the partial message (streaming).
Delta *ResponseMessage `json:"delta,omitempty"`
// FinishReason indicates why completion stopped.
FinishReason string `json:"finish_reason"`
}
ResponseChoice represents a single completion choice.
type ResponseMessage ¶
ResponseMessage represents a message in a completion choice.
type ResponsesError ¶ added in v0.0.6
type ResponsesExtractor ¶ added in v0.0.6
type ResponsesExtractor struct{}
func NewResponsesExtractor ¶ added in v0.0.6
func NewResponsesExtractor() *ResponsesExtractor
func (*ResponsesExtractor) Extract ¶ added in v0.0.6
func (e *ResponsesExtractor) Extract(resp *http.Response) (llmproxy.ResponseMetadata, []byte, error)
type ResponsesInputDetails ¶ added in v0.0.6
type ResponsesInputDetails struct {
CachedTokens int `json:"cached_tokens,omitempty"`
}
type ResponsesOutputAnnotation ¶ added in v0.0.7
type ResponsesOutputContent ¶ added in v0.0.6
type ResponsesOutputContent struct {
Type string `json:"type"`
Text string `json:"text,omitempty"`
Annotations []ResponsesOutputAnnotation `json:"annotations,omitempty"`
Logprobs interface{} `json:"logprobs,omitempty"`
}
type ResponsesOutputDetails ¶ added in v0.0.6
type ResponsesOutputDetails struct {
ReasoningTokens int `json:"reasoning_tokens,omitempty"`
}
type ResponsesOutputItem ¶ added in v0.0.6
type ResponsesOutputItem struct {
ID string `json:"id"`
Type string `json:"type"`
Status string `json:"status"`
Role string `json:"role,omitempty"`
Content []ResponsesOutputContent `json:"content,omitempty"`
Name string `json:"name,omitempty"`
Arguments string `json:"arguments,omitempty"`
Summary []ResponsesOutputSummary `json:"summary,omitempty"`
}
type ResponsesOutputSummary ¶ added in v0.0.7
type ResponsesParser ¶ added in v0.0.6
type ResponsesParser struct{}
func (*ResponsesParser) Parse ¶ added in v0.0.6
func (p *ResponsesParser) Parse(body io.ReadCloser) (llmproxy.BodyMetadata, []byte, error)
type ResponsesRequest ¶ added in v0.0.6
type ResponsesRequest struct {
Model string `json:"model"`
Input interface{} `json:"input,omitempty"`
Instructions string `json:"instructions,omitempty"`
MaxOutputTokens int `json:"max_output_tokens,omitempty"`
Temperature *float64 `json:"temperature,omitempty"`
TopP *float64 `json:"top_p,omitempty"`
Stream bool `json:"stream"`
Tools []interface{} `json:"tools,omitempty"`
Truncation string `json:"truncation,omitempty"`
Custom map[string]interface{} `json:"-"`
}
func (*ResponsesRequest) UnmarshalJSON ¶ added in v0.0.6
func (r *ResponsesRequest) UnmarshalJSON(data []byte) error
type ResponsesResponse ¶ added in v0.0.6
type ResponsesResponse struct {
ID string `json:"id"`
Object string `json:"object"`
Created int64 `json:"created"`
Model string `json:"model"`
Status string `json:"status"`
Output []ResponsesOutputItem `json:"output"`
Usage ResponsesUsage `json:"usage"`
Error *ResponsesError `json:"error,omitempty"`
}
type ResponsesStreamingExtractor ¶ added in v0.0.8
type ResponsesStreamingExtractor struct {
*ResponsesExtractor
}
func NewResponsesStreamingExtractor ¶ added in v0.0.8
func NewResponsesStreamingExtractor() *ResponsesStreamingExtractor
func (*ResponsesStreamingExtractor) ExtractStreamingWithController ¶ added in v0.0.8
func (e *ResponsesStreamingExtractor) ExtractStreamingWithController(resp *http.Response, w http.ResponseWriter, rc *http.ResponseController) (llmproxy.ResponseMetadata, error)
func (*ResponsesStreamingExtractor) IsStreamingResponse ¶ added in v0.0.8
func (e *ResponsesStreamingExtractor) IsStreamingResponse(resp *http.Response) bool
type ResponsesUsage ¶ added in v0.0.6
type ResponsesUsage struct {
InputTokens int `json:"input_tokens"`
OutputTokens int `json:"output_tokens"`
TotalTokens int `json:"total_tokens"`
InputTokensDetails *ResponsesInputDetails `json:"input_tokens_details,omitempty"`
OutputTokensDetails *ResponsesOutputDetails `json:"output_tokens_details,omitempty"`
}
type StreamingExtractor ¶ added in v0.0.6
type StreamingExtractor struct {
*Extractor
}
func NewStreamingExtractor ¶ added in v0.0.6
func NewStreamingExtractor() *StreamingExtractor
func (*StreamingExtractor) ExtractStreamingWithController ¶ added in v0.0.6
func (e *StreamingExtractor) ExtractStreamingWithController(resp *http.Response, w http.ResponseWriter, rc *http.ResponseController) (llmproxy.ResponseMetadata, error)
func (*StreamingExtractor) IsStreamingResponse ¶ added in v0.0.6
func (e *StreamingExtractor) IsStreamingResponse(resp *http.Response) bool
type StreamingMultiAPIExtractor ¶ added in v0.0.6
type StreamingMultiAPIExtractor struct {
*MultiAPIExtractor
// contains filtered or unexported fields
}
func NewStreamingMultiAPIExtractor ¶ added in v0.0.6
func NewStreamingMultiAPIExtractor() *StreamingMultiAPIExtractor
func (*StreamingMultiAPIExtractor) ExtractStreamingWithController ¶ added in v0.0.6
func (e *StreamingMultiAPIExtractor) ExtractStreamingWithController(resp *http.Response, w http.ResponseWriter, rc *http.ResponseController) (llmproxy.ResponseMetadata, error)
func (*StreamingMultiAPIExtractor) IsStreamingResponse ¶ added in v0.0.6
func (e *StreamingMultiAPIExtractor) IsStreamingResponse(resp *http.Response) bool
type UsageInfo ¶
type UsageInfo struct {
PromptTokens int `json:"prompt_tokens"`
CompletionTokens int `json:"completion_tokens"`
TotalTokens int `json:"total_tokens"`
PromptTokensDetails *PromptTokensDetails `json:"prompt_tokens_details,omitempty"`
CompletionTokensDetails *CompletionTokensDetails `json:"completion_tokens_details,omitempty"`
}
UsageInfo tracks token usage in an OpenAI-compatible response.