llm

type ChatOpts ¶

type ChatOpts struct {
	ToolChoice     string
	TraceID        string
	Grammar        string
	ResponseFormat any
}

ChatOpts carries per-call switches.

ToolChoice: "", "auto", "required", or a JSON-encoded

{"type":"function","function":{"name":"foo"}}.
OpenAI-compatible providers honor "required" or a specific
tool spec.

TraceID: the value sent in the X-Trace-Id header, for correlation

only — server-side logs can attribute requests back to a thread.
Scheduling priority is keyed off the API key (Authorization
Bearer), not this header. Harness Sessions set TraceID = thread
id; empty is fine (the header just won't be sent).

Grammar: when non-empty, forwarded as the request body's "grammar"

field — a GBNF grammar the server constrains token sampling to
(llama.cpp / corrallm). Raw passthrough; the server owns the
syntax. Use for hard structural guarantees the model cannot
violate (vs the client-side agent.Validator fix loop, which
corrects a bad reply after the fact).

ResponseFormat: when non-nil, forwarded as the request body's

"response_format" — e.g. map[string]any{"type":"json_object"} or a
{"type":"json_schema","json_schema":{...}} object. Marshaled
as-is; the server decides support.

Nil opts behaves as the default.

type Client ¶

type Client struct {

	// RetryBudget caps the total wall-clock a single request spends retrying
	// 429/5xx before giving up (exponential backoff to retryMaxBackoff,
	// honoring retry_after, but bounded). 0 → defaultRetryBudget (5m). The
	// caller's ctx deadline still wins if shorter. Set it per Client for a
	// busy endpoint.
	RetryBudget time.Duration
	// contains filtered or unexported fields
}

Client sends requests to an OpenAI-compatible LLM endpoint.

func NewClient(baseURL, apiKey, model string) *Client

func (c *Client) Chat(ctx context.Context, messages []Message, tools []ToolDef) (string, []ToolCall, error)

func (c *Client) ChatStream(ctx context.Context, messages []Message, tools []ToolDef, opts *ChatOpts) (<-chan StreamChunk, error)

type Message ¶

type Message struct {
	Role    string `json:"role"`
	Content string `json:"content"`
	Name    string `json:"name,omitempty"`
}

Message represents a chat message in the OpenAI-compatible format.

type Session ¶

type Session struct {
	ID       string
	Messages []Message
	// contains filtered or unexported fields
}

Session manages the message history and tool configuration for one LLM conversation.

func NewSession(id, system string) *Session

func (s *Session) AddAssistantMessage(content string)

func (s *Session) AddToolCalls(calls []ToolCall)

func (s *Session) AddToolMessage(toolCallID, content string)

func (s *Session) AddToolResult(toolCallID, result string)

func (s *Session) AddUserMessage(content string)

func (s *Session) ChatStream(ctx context.Context, client *Client) (<-chan StreamChunk, error)

func (s *Session) GetMessages() []Message

func (s *Session) GetTools() []ToolDef

func (s *Session) SetTools(tools []ToolDef)

func (s *Session) String() string

func (s *Session) TrimMessages(max int)

type StreamChunk ¶

type StreamChunk struct {
	Content  string
	ToolCall *ToolCall
	Done     bool
	Error    string
	Usage    *Usage
}

StreamChunk is one token from the streaming response. The final chunk carries Usage (when the provider supports include_usage); all other fields will be zero on that chunk.

type ToolCall ¶

type ToolCall struct {
	ID       string `json:"id"`
	Type     string `json:"type"`
	Function struct {
		Name      string `json:"name"`
		Arguments string `json:"arguments"`
	} `json:"function"`
}

ToolCall represents a tool call request from the LLM.

type ToolDef ¶

type ToolDef struct {
	Type     string `json:"type"`
	Function struct {
		Name        string `json:"name"`
		Description string `json:"description"`
		Parameters  any    `json:"parameters"`
	} `json:"function"`
}

ToolDef describes a tool the LLM can call, matching the OpenAI tool format.

type Usage ¶

type Usage struct {
	PromptTokens             int `json:"prompt_tokens"`
	CompletionTokens         int `json:"completion_tokens"`
	TotalTokens              int `json:"total_tokens"`
	CacheReadInputTokens     int `json:"cache_read_input_tokens,omitempty"`
	CacheCreationInputTokens int `json:"cache_creation_input_tokens,omitempty"`
}

Usage carries provider-reported token counts for one chat completion. Fields beyond prompt/completion/total are optional and only filled when the provider returns them (Anthropic's cache fields, OpenAI's reasoning tokens, etc).