llm

package

v0.239.0-rc.2 Latest Latest Go to latest Published: Oct 28, 2026 License: Apache-2.0 Imports: 10 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/inference-gateway/cli

Links

Open Source Insights

Documentation ¶

Index ¶

Variables
type Client
- func New(client sdk.Client) *Client
- func (c *Client) Complete(ctx context.Context, req Request) (Response, error)
type Completer
type Request
type Response
- func (r Response) Answer() string

Constants ¶

This section is empty.

Variables ¶

View Source

var ErrEmptyResponse = errors.New("model returned an empty response")

ErrEmptyResponse marks a model that finished without any text at all.

View Source

var ErrTokenBudgetExhausted = errors.New("token budget exhausted")

ErrTokenBudgetExhausted marks a model that spent max_tokens before answering.

Functions ¶

This section is empty.

Types ¶

type Client ¶ added in v0.236.2

type Client struct {
	// contains filtered or unexported fields
}

Client is the Completer over an SDK client.

func New ¶ added in v0.236.2

func New(client sdk.Client) *Client

func (*Client) Complete ¶ added in v0.236.2

func (c *Client) Complete(ctx context.Context, req Request) (Response, error)

Complete runs the request as a stream, so a reasoning model's first bytes arrive before the gateway's header timeout, and folds the chunks back into one Response. The SDK's reader goroutine exits with the context, so Complete owns a cancel for it.

type Completer ¶ added in v0.236.2

type Completer interface {
	Complete(ctx context.Context, req Request) (Response, error)
}

Completer answers one side call. The judge, insights and custom shortcuts consume it, so their tests script an answer without a stream.

type Request ¶ added in v0.236.2

type Request struct {
	Model     string
	System    string
	Prompt    string
	MaxTokens int
	Progress  io.Writer
}

Request is one no-tools side call: an optional system prompt, one user prompt and a token budget. Progress, when set, receives a status line while the model works. The caller owns the deadline via ctx.

type Response ¶ added in v0.236.2

type Response struct {
	Content   string
	Reasoning string
	Usage     *sdk.CompletionUsage
}

Response is what the model answered, split the way the API splits it.

func (Response) Answer ¶ added in v0.236.2

func (r Response) Answer() string

Answer is the trimmed content, or the reasoning when a reasoning model left its whole answer there and nothing in content.

Source Files ¶

View all Source files

llm.go

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL