api

package
v0.0.0-...-a40ddb2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 10, 2026 License: MIT Imports: 18 Imported by: 0

Documentation

Overview

Package api provides higher level functions to wrap the OpenAI chat completions API.

Index

Constants

This section is empty.

Variables

View Source
var (
	// Optional logging of raw JSON requests and responses
	TraceRequests = false
	TraceStream   = false
	TraceTo       = os.Stderr
)
View Source
var (
	// Used by DefaultConfig
	DefaultSystemMessage = "You are a helpful assistant. You should answer concisely unless more detail is requested. The current date is {{today}}."
	// Used by NewRequest
	ParallelToolCalls = true
)

Functions

func ChatCompletionToolParams

func ChatCompletionToolParams(tools []ToolFunction) (params []openai.ChatCompletionToolUnionParam)

Tool parameters for given list of tools

func FromMessage

func FromMessage(m Message, reasoningField string) openai.ChatCompletionMessageParamUnion

Convert our message format to openai chat completion struct

func GetContent

func GetContent(raw string) (content, reasoning string)

Get content and reasoning content from raw JSON message

func MaxModelLength

func MaxModelLength(server Server, baseURL string) (maxLen int, err error)

Get max content length for current model

func Pretty

func Pretty(v any) string

Pretty print struct

func Tokenize

func Tokenize(server Server, baseURL string, messages []openai.ChatCompletionMessageParamUnion) (numTokens int, err error)

Convert text to tokens for current model

Types

type Accumulator

type Accumulator struct {
	openai.ChatCompletionAccumulator
	Content   string
	Reasoning string
	// contains filtered or unexported fields
}

type CallbackFunc

type CallbackFunc func(channel, content string, index int, end bool)

Generated content where channel is either analysis (i.e. reasoning text), final (generated text) or tool (response from tool call) and index is the count of the message on the current stream, end is set on final message completion and full rather than delta content is sent

type Client

type Client struct {
	openai.Client
	Server         Server
	BaseURL        string
	ModelName      string
	ReasoningField string
	ContextLength  int
}

Client and associated info

func NewClient

func NewClient(server Server, modelName string, opts ...option.RequestOption) (c Client, err error)

Create new client with default settings if no options are given. If set then will use OPENAI_BASE_URL and OPENAI_API_KEY environment variables. The model name is optional for LlamaCPP and LLLM - they will use the currently loaded model.

func (*Client) ChatCompletion

func (c *Client) ChatCompletion(ctx context.Context, request Conversation, callback CallbackFunc, statsCallback func(Stats), tools ...ToolFunction) ([]Message, error)

Chat completion without streaming with optional function call support. The list of new generated messages are returned. callback and statsCallback are called after each stage of generation - i.e. reasoning text, tool response and final response.

func (*Client) ChatCompletionStream

func (c *Client) ChatCompletionStream(ctx context.Context, request Conversation, callback CallbackFunc, statsCallback func(Stats), tools ...ToolFunction) ([]Message, error)

As per ChatCompletion but will stream responses as they are generated.

func (*Client) CompactMessages

func (c *Client) CompactMessages(conv Conversation, limit int) error

Estimate number of prompt tokens generated from the given list of messages. If this exceeds the given limit then the oldest messages in the conversation are marked as excluded.

func (*Client) NewRequest

func (c *Client) NewRequest(modelName string, conv Conversation, tools ...ToolFunction) openai.ChatCompletionNewParams

Create a new chat completion request with given config settings. Messages with Excluded set are omitted from the request. Includes reasoning content starting from the beginning of the latest turn.

type Config

type Config struct {
	SystemPrompt      string       `json:"system_prompt"`
	ReasoningEffort   string       `json:"reasoning_effort"` // low | medium | high | none
	Tools             []ToolConfig `json:"tools,omitzero"`
	Temperature       float64      `json:"temperature,omitzero"`
	TopP              float64      `json:"top_p,omitzero"`
	TopK              int          `json:"top_k,omitzero"`
	PresencePenalty   float64      `json:"presence_penalty,omitzero"`
	RepetitionPenalty float64      `json:"repetition_penalty,omitzero"`
	CompactThreshold  float64      `json:"compact_threshold,omitzero"` // if set then apply message compaction if hit this fraction of model context length
}

func DefaultConfig

func DefaultConfig(tools ...ToolFunction) Config

Get default configuration with given tools enabled

type Conversation

type Conversation struct {
	ID        string                     `json:"id"`
	Config    Config                     `json:"config"`
	Messages  []Message                  `json:"messages"`
	NumTokens int                        `json:"num_tokens"`
	ToolData  map[string]json.RawMessage `json:"tool_data,omitzero"`
}

func NewConversation

func NewConversation(cfg Config) Conversation

Create a new conversation and assign unique ID

func (Conversation) LastUserMessageNumber

func (c Conversation) LastUserMessageNumber() int

Most recent non-excluded user message number or -1 if not found

type Item

type Item struct {
	ID      string `json:"id"`
	Summary string `json:"summary"`
}

type Message

type Message struct {
	Role            string          `json:"role"`            // user | assistant | tool
	Update          bool            `json:"update,omitzero"` // true if update to existing message
	End             bool            `json:"end,omitzero"`    // true if update and message is now complete
	Content         string          `json:"content"`
	Reasoning       string          `json:"reasoning,omitzero"`
	ToolCall        json.RawMessage `json:"tool_call,omitzero"`
	ToolCallID      string          `json:"tool_call_id,omitzero"`
	ContentTokens   int             `json:"content_tokens,omitzero"`
	ReasoningTokens int             `json:"reasoning_tokens,omitzero"`
	Excluded        bool            `json:"excluded,omitzero"` // message is ignored by NewRequest if this is set
}

func ToMessage

func ToMessage(m openai.ChatCompletionMessageParamUnion, reasoningField string) Message

Convert from openai chat completion message to our message format

type Request

type Request struct {
	Action  string  `json:"action"`           // add | list | load | delete | config
	ID      string  `json:"id,omitzero"`      // if action=load,delete uuid format
	Message Message `json:"message,omitzero"` // if action=add
	Config  *Config `json:"config,omitzero"`  // if action=config
}

Chat API request from frontend to webserver

type Response

type Response struct {
	Action       string       `json:"action"`                // add | list | load | config | stats
	Message      Message      `json:"message,omitzero"`      // if action=add
	Conversation Conversation `json:"conversation,omitzero"` // if action=load
	List         []Item       `json:"list,omitzero"`         // if action=list
	Config       Config       `json:"config,omitzero"`       // if action=config
	Stats        Stats        `json:"stats,omitzero"`        // if action=stats
}

Chat API response from webserver back to frontend

type Server

type Server int

Endpoint to connect to

const (
	LlamaCPP   Server = 0
	VLLM       Server = 1
	OpenRouter Server = 2
	Cerebras   Server = 3
)

func GetServer

func GetServer() Server

Get current server from LLM_SERVER env if set

func (Server) String

func (i Server) String() string

type Stats

type Stats struct {
	Model            string         `json:"model"`             // model name
	ApiCalls         int            `json:"api_calls"`         // total number of API calls
	ApiTime          int            `json:"api_time"`          // total elapsed time in API calls in msec
	CompletionTokens int            `json:"completion_tokens"` // no. of completion tokens generated
	PromptTokens     int            `json:"prompt_tokens"`     // max prompt length
	ToolCalls        int            `json:"tool_calls"`        // total number of tool calls
	Functions        map[string]int `json:"functions"`         // numer of tool calls by function name
	ToolTime         int            `json:"tool_time"`         // total elapsed time in tool calls in msec
}

func (*Stats) APICallInfo

func (s *Stats) APICallInfo() string

func (*Stats) CompletionTokensPerSec

func (s *Stats) CompletionTokensPerSec() float64

func (*Stats) Loginfo

func (s *Stats) Loginfo()

func (*Stats) ToolCallInfo

func (s *Stats) ToolCallInfo() string

type ToolConfig

type ToolConfig struct {
	Name    string `json:"name"`
	Enabled bool   `json:"enabled"`
}

type ToolFunction

type ToolFunction interface {
	// function name and argument schema
	Definition() shared.FunctionDefinitionParam
	// call with args in JSON format - if err is set it is treated as fatal, else the model can retry with new args
	Call(args string) (req, resp string, err error)
}

Interface implented by tools which can be called by the model.

Directories

Path Synopsis
browser
Package browser implements an OpenAI compatible web browser tool plugin which uses the brave search API and firecrawl scrape API.
Package browser implements an OpenAI compatible web browser tool plugin which uses the brave search API and firecrawl scrape API.
python
Python tool to execute code in a docker container
Python tool to execute code in a docker container
weather
Package weather implements tool functions to call the openweathermap API.
Package weather implements tool functions to call the openweathermap API.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL