mllm

package
v0.3.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 4, 2026 License: MIT Imports: 17 Imported by: 0

README

mllm

tiny ai client

Environment Runtime Config

Reads these env vars (in order):

Model: "MLLM_MODEL", "OLLAMA_MODEL", "LLM_MODEL"
Host: "MLLM_HOST", "OLLAMA_HOST", "LLM_HOST"
API Key (optional): "MLLM_API_KEY", "OLLAMA_API_KEY", "LLM_API_KEY"

Use DEBUG_LLM=1 to see json requests/responses

Provider

Tested with llama.cpp:

export LLM_MODEL=gemma-4-E2B-it-Q4_K_M LLM_HOST=http://127.0.0.1:9931/v1/chat/completions LLM_API_KEY=some-key-here

Tested with ollama:

export LLM_MODEL=gemma4:26b LLM_HOST=http://127.0.0.1:11434/v1/chat/completions

Usage

Typical usage for TUI would be (background) streaming with a custom StreamHandler to write into a view that renders markdown.

Background

Use StreamBG to send the request, returning a chan error that is closed when stream is finished. A nil error is sent if no error occurred. The channel must be emptied.

TUI/GUI applications may use StreamBG with custom handlers to stream the LLM output frames directly to a view (possibly rendering markdown at intervals), and can access cr.Done to signal other threads that the generation is finished.

 client := mllm.Default() // uses env for Model and Host
 think := false
 // ChatResponse itself has a Handler method that is a StreamHandler func
 var chatResp mllm.ChatResponse
 var handlers []StreamHandler // stream handlers are called each frame/delta
 handlers = []StreamHandler{
   mllm.NewHandlerOut(os.Stderr, "THINKING:\n", "\nFINISHED THINKING\n"),
   chatResp.Handler,
   // your other handler
 }

 err := <- e.StreamBG(ctx, messages, think, handlers...)
 // then, give chatResp to toolrunner, etc

See also: test file

Foreground

For non-streaming behavior, use msg, err := client.Get(...)

Under the hood, we still use "stream": true, but our GetMessage function itself does not return until the llm is finished (same with Get function)

 client := mllm.Default() // uses env for Model and Host
 think := false
 var resp mllm.Message
 var err error
 cr, err := client.Get(ctx, messages, think)
 resp = cr.Message // essentially the same as GetMessage
 // or:
 resp, err = client.GetMessage(ctx, messages, think)
}

See also: test file

StreamHandler

Although it is usually not necessary, a new StreamHandler is easy to make. It might contain state. If it returns any error the stream will cut short. A StreamHandler is called very quickly, and will lag the llm stream (see example below).

import "github.com/aerth/mostly/mllm"

// as a function

func Dummy(cr mllm.ChatResponse) error {
 return nil
}

// as a method, stateful for adding each frame to it's own message copy
type X struct{Message mllm.Message, Ch chan mllm.Message}

func (x *X) Dummy(cr mllm.ChatResponse) error {
  x.Message = x.Message.Add(cr.Message) // combined
  if cr.Done {
 select {
 case x.Ch <- x.Message:
 default:
 return fmt.Errorf("GUI message channel is full")
 }
  }
 return nil
}
Tools

In this package, a Tool contains an "Exec" function (who's signature is up to you), paired with a JSON "Spec" which is a json.RawMessage. A tool Call, given by LLM, has "Arguments", (which may be an object or stringified object, or null or empty string). This package does not run tools, or care what function signature you use as tool's Exec, and therefore you must implement your own "tool runner".

An example tool runner is in cmd/example

See also: tool test file

Tool Exec Funcs

Example tool function signatures might include (but are in no way limited to):

func(ctx context.Context, args json.RawMessage) (string, error)

func(args map[string]any) (any, error)

func(ctx context.Context, args SomeArgumentType, other SessionData) (SomeResponseType, error)

call.ReplyTool

The call.ReplyTool helper function (for a tool runner) to wrap a tool output into text for a new message to be added to the conversation. It adds ToolCallID and ToolName, and JSON encodes the input for the new message's Content.

Example Tool

Example TimeTool implemented in Go, using a (ctx context.Context, args map[string]any) (string, error) exec func signature.

The "function": wrapper is unnecessary as this package always ensures the spec is wrapped.

var TimeTool = mllm.Tool{
 Spec: timeSpec,
 Exec: getTime,
}

var timeSpec = json.RawMessage(`{
  "type": "function",
  "function": {
    "name": "get_time",
    "description": "get current time",
    "parameters": {
      "type": "object",
      "properties": {
        "zone": {
          "type": "string"
        }
      },
      "required": ["zone"]
    }
  }
}`)

func getTime(ctx context.Context, args map[string]any) (string, error) {
 zone, ok := args["zone"].(string)
 if !ok {
  return "", errors.New("invalid zone")
 }
 loc, err := time.LoadLocation(zone)
 if err != nil {
  return "", fmt.Errorf("load location failed: %w", err)
 }
 return time.Now().In(loc).Format(time.RFC3339), nil
}

Documentation

Overview

Package mllm is simple llm client, part of aerth/mostly project

Index

Constants

This section is empty.

Variables

View Source 
var (
	DefaultThinkPre         = "Thinking:\n"
	DefaultThinkPost        = "\n... done thinking.\n"
	ThinkRequestType        = false
	StopAtFirstHandlerError = true
	ErrNoLines              = errors.New("no lines read")
	HideThink               = false
	HideTools               = false
	EnvModel                = []string{"MLLM_MODEL", "OLLAMA_MODEL", "LLM_MODEL"}
	EnvHost                 = []string{"MLLM_HOST", "OLLAMA_HOST", "LLM_HOST"}
	EnvKey                  = []string{"MLLM_API_KEY", "OLLAMA_API_KEY", "LLM_API_KEY"}
	DebugLLM2               = os.Getenv("DEBUG_LLM") == "2"
	DebugLLM                = os.Getenv("DEBUG_LLM") == "1" || DebugLLM2
	ErrNoConvo              = errors.New("no messages in conversation")
	Quiet                   = false // quiet the debug
)
View Source 
var (
	DefaultOptions = Options{
		Temperature: new(1.0),
		Seed:        -1,
	}
	ForcePath = false
)
View Source 
var (
	Errorf = stackerr.Errorf
	Slog   = slog.Default
)
View Source 
var Default = func() E {
	e := NewEndpoint(nil, nil)
	return e
}

Default creates a new reusable LLM endpoint from env using NewEndpoint with no tools no options.

View Source 
var DefaultChatMessageFormatter = func(cm ChatMessage) ([]Message, bool) {
	return []Message{cm.Message}, true
}

DefaultChatMessageFormatter pass through ChatMessageFormatter In a custom implementation, this might add text to Content, replace Role, or insert messages before/after.

View Source 
var DefaultEmbedOptions map[string]any // nil
View Source 
var DefaultThread = uuid.Nil()

DefaultThread for all Message.ToChatMessage calls.

Applications may use this for chat rooms, conversations, etc

View Source 
var DefaultWriter io.Writer = os.Stderr

DefaultWriter added to all new endpoints

View Source 
var EmbedOptions = struct {
	URL        url.URL
	Model      string // if req.Model is empty and env is empty
	Dimensions int
	EnvHost    string // the ENV_VAR to check to override [E.Host] (for when using 2 different hosts for llm,embeddings.)
	Path       string // custom endpoint url path (default /api/embed)
}{
	Model:   "nomic-embed-text:v1.5",
	EnvHost: "LLM_EMBED_HOST",
	Path:    "/v1/embeddings",
}

EmbedOptions application-wide global options

Apart from Path, all serve as fallback if not present in request/env.

View Source 
var NewE = NewEndpoint

NewE custom NewEndpoint

View Source 
var ToolWriter = toolWriter

Functions

func Noop added in v0.3.2 

func Noop(cr ChatResponse) error

func StderrHandler 

func StderrHandler(cr ChatResponse) error

StderrHandler plain 🍔

func StdoutHandler 

func StdoutHandler(cr ChatResponse) error

StdoutHandler plain 🍔

func Summary 

func Summary(w io.Writer, st Stats) (err error)

Summary in well-known format

func ToMarshalers added in v0.3.2 

func ToMarshalers(v ...json.Marshaler) []json.Marshaler

func WrapToolSpec added in v0.3.2 

func WrapToolSpec(js json.RawMessage) ([]byte, error)

Types

type Call added in v0.3.1 

type Call[T any] struct {
	ID       string      `json:"id,omitempty"`
	Index    int         `json:"index"`
	Type     string      `json:"type,omitempty"`
	Function Function[T] `json:"function"`
}

Call is for a tool-runner to decode arguments. Callers may use Call[struct{Path string}] to decode `"arguments": {"path": "foo.txt"}`

It handles the case where Arguments json object is passed as a json string.

See Call.UnmarshalJSON

func (*Call[T]) FromPlain added in v0.3.2 

func (c *Call[T]) FromPlain(from Call[json.RawMessage]) error

func (*Call[T]) FromRaw added in v0.3.2 

func (c *Call[T]) FromRaw(js json.RawMessage) error

func (Call[T]) ReplyTool added in v0.3.2 

func (c Call[T]) ReplyTool(val any) Message

func (Call[T]) ReplyToolOrError added in v0.3.2 

func (c Call[T]) ReplyToolOrError(val any, err error) Message

func (*Call[T]) ToJSON added in v0.3.2 

func (c *Call[T]) ToJSON() json.RawMessage

func (Call[T]) ToRaw added in v0.3.2 

func (c Call[T]) ToRaw() Call[json.RawMessage]

func (*Call[T]) UnmarshalJSON added in v0.3.1 

func (c *Call[T]) UnmarshalJSON(b []byte) error

type CallFuncV1 added in v0.3.2 

type CallFuncV1 = func(ctx context.Context, args map[string]any) (output any, err error)

CallFuncV1 is completly optional and any may be used by a custom toolrunner

type CallFuncV1B added in v0.3.2 

type CallFuncV1B = func(ctx context.Context, args json.RawMessage) (output any, err error)

CallFuncV1 is completly optional and any may be used by a custom toolrunner

type Calls added in v0.3.2 

type Calls[T any] []Call[T]

func (Calls[T]) ToJSON added in v0.3.2 

func (c Calls[T]) ToJSON() []json.RawMessage

type ChatHandlerT 

type ChatHandlerT interface {
	HandleChat(cr ChatResponse) error
}

type ChatMessage 

type ChatMessage struct {
	MessageID uuid.UUID `gorm:"type:uuid;uniqueKey;default:null;" json:"message_id,omitempty"` // MessageID is unique identifier for this message
	Message             // Base Message
	From      string    `json:"from,omitempty" gorm:"default:null"`             // The name of user/bot who created this message
	CreatedAt time.Time `json:"created_at"`                                     // Created or received at
	Thread    uuid.UUID `json:"thread,omitempty" gorm:"type:uuid;default:null"` // Conversation thread this message belongs to
}

ChatMessage is a Message but with additional metadata (for a multi-participant conversation)

Is a gorm model, (without uint ID)

ChatMessage API is subject to change and not stablized yet

func NewChatMessage 

func NewChatMessage(m Message, from string, thread uuid.UUID) ChatMessage

NewChatMessage from (see also Message.ToChatMessage)

type ChatMessageFormatter 

type ChatMessageFormatter = func(cm ChatMessage) ([]Message, bool)

ChatMessageFormatter may skip a message by returning false

The returned formatted messages are NEVER stored only sent

type ChatMessageWithID added in v0.3.2 

type ChatMessageWithID struct {
	ID uint `json:"-" gorm:"primaryKey"` // TODO: cant json id here for [Message2.ToolCallID2] llama.cpp
	ChatMessage
}

type ChatMessages 

type ChatMessages []ChatMessage

ChatMessages like Messages but with additional metadata (multiple multi-participant conversations)

func (ChatMessages) ToMessages 

func (cm ChatMessages) ToMessages() Messages

func (ChatMessages) ToMessagesFormat 

func (cm ChatMessages) ToMessagesFormat(fn ChatMessageFormatter) []Message

ToMessagesFormat returns a **temporary** convo for llm to produce a single new Message The returned convo should NEVER be stored (only sent to llm) as the Content is modified by formatter.

func (ChatMessages) Top added in v0.2.5 

func (cm ChatMessages) Top() ChatMessage

func (ChatMessages) TopN added in v0.2.5 

func (cm ChatMessages) TopN(n int) ChatMessages

func (ChatMessages) TopRole added in v0.2.5 

func (cm ChatMessages) TopRole(role string) ChatMessage

type ChatRequest 

type ChatRequest struct {
	Model     string           `json:"model,omitempty"`
	Messages  []Message        `json:"messages,omitempty"`
	Stream    bool             `json:"stream"`
	Think     *Think           `json:"reasoning_effort,omitempty"` // ollama endpoint "think" but /v1/chat/completions is "thinking"
	KeepAlive int              `json:"keep_alive,omitempty"`
	Tools     []json.Marshaler `json:"tools,omitempty"`
	*Options  `gorm:"type:jsonb;serializer:json;default:null"`
}

ChatRequest sent to LLM (/v1/chat/completions or compatible)

type ChatResponse 

type ChatResponse struct {
	LLMID string `json:"id,omitempty" gorm:"-"`
	//	Object string
	Model       string          `json:"model"`
	Created     json.RawMessage `json:"created,omitempty"`
	Done        bool            `json:"done"` // for stream
	DoneReason  string          `json:"done_reason,omitempty"`
	DoneReason2 string          `json:"finish_reason,omitempty" gorm:"-"` // INTERNAL do not use
	Message     Message         `json:"message"`                          // for stream, handlers
	Stats       `gorm:"embedded"`
	Choices     []Choice `json:"choices,omitempty"` // INTERNAL do not use
}

ChatResponse streamable has ChatResponse.Handler

See LMessage

Example: 

var cr mllm.ChatResponse
err := <-mllm.Default().StreamBG(ctx, mllm.Messages{{Role: "user", Content: "hi"}}, false, cr.Handler)
println("Bot:", cr.Message.Content)

func (*ChatResponse) Handler 

func (c *ChatResponse) Handler(in ChatResponse) error

Handler for E.StreamBG and E.Stream

Is a StreamHandler to gather Message

func (ChatResponse) SummaryString 

func (c ChatResponse) SummaryString() string

SummaryString collects [ChatResponse.Stats] into a string

type Choice added in v0.3.1 

type Choice struct {
	Index        int     `json:"index"`
	FinishReason *string `json:"finish_reason,omitempty"`
	Delta        Message `json:"delta"`
}

type E 

type E struct {
	Host  url.URL // the LLM API (required)
	Model string  // the model name (required)

	Client  *http.Client      `json:"-"` // custom http client (optional)
	Headers map[string]string `json:"-"` // custom headers (optional)
	Tools   []json.Marshaler  // Bring your own tools (optional)
	Options *Options          // Options (optional) added to each NewRequest

	Writer       io.Writer // used in [E.Get] as StreamWriter if no StreamWriter
	StreamWriter W1        `json:"-" gorm:"-"` // used as only handler **IF no handlers given**

	PreRequest func(request *ChatRequest) error `json:"-"`

	// Handlers **added** to all requests
	Handlers []StreamHandler `json:"-"`
}

E (Endpoint) holds information for future requests

Writer for streaming output with ThinkPre and ThinkPost.

func NewEndpoint 

func NewEndpoint(options *Options, tools []json.Marshaler) E

NewEndpoint reads env and returns a new reusable Endpoint

See E.StreamBG or E.Stream or E.Get

func (E) AddHandlers 

func (e E) AddHandlers(h ...StreamHandler) E

func (E) Chat added in v0.2.5 

func (e E) Chat(ctx context.Context, messages ChatMessages, think any) (ChatMessage, error)

func (E) Do added in v0.2.3 

func (e E) Do(ctx context.Context, creq ChatRequest, handlers []StreamHandler) error

Do sends a ChatRequest and streams response in E.Read

func (E) Embed added in v0.2.2 

func (e E) Embed(ctx context.Context, req EmbedRequest) (*EmbedResponse, error)

Embed generates new embeddings

See NewNomicDocument and NewNomicQuery for model-specific query prefixes

func (E) Get 

func (e E) Get(ctx context.Context, messages []Message, think any) (ChatResponse, error)

Get calls E.Stream and waits until finished, returning the full ChatResponse

Uses E.Stream, writes stream text to [E.Writer] if non-nil, returns full ChatResponse

func (E) GetMessage added in v0.3.1 

func (e E) GetMessage(ctx context.Context, messages []Message, think any) (Message, error)

func (E) NewRequest added in v0.2.3 

func (e E) NewRequest(messages []Message, think any) *ChatRequest

NewRequest creates a ChatRequest adds e.Model, e.Tools, and e.Options

func (E) Read added in v0.2.2 

func (e E) Read(reader *bufio.Reader, handlers ...StreamHandler) error

Read a stream, collecting for StreamHandlers

func (E) Stream 

func (e E) Stream(ctx context.Context, messages []Message, think any, handlers ...StreamHandler) error

Stream with handlers(only) or to E.Writer, calls E.Do

func (E) StreamBG 

func (e E) StreamBG(ctx context.Context, messages []Message, think any, handlers ...StreamHandler) chan error

StreamBG uses E.Stream, closing ch after streaming is finished, sending non-nil error to the channel

func (E) ValidateConfig 

func (e E) ValidateConfig(think any) error

func (E) WithHandlers 

func (e E) WithHandlers(h ...StreamHandler) E

func (E) WithOptions 

func (e E) WithOptions(options *Options) E

func (E) WithTools 

func (e E) WithTools(tools ...json.Marshaler) E

func (E) WithoutTools added in v0.2.1 

func (e E) WithoutTools() E

type EmbedRequest added in v0.2.2 

type EmbedRequest struct {
	Model     string `json:"model"`
	Input     string `json:"input"`
	KeepAlive int    `json:"keep_alive,omitempty"`
	// Dimensions truncates the output embedding to the specified dimension.
	Dimensions int   `json:"dimensions,omitempty"`
	Truncate   *bool `json:"truncate,omitempty"`
	// Options lists model-specific options.
	Options map[string]any `json:"options,omitempty"`

	Host url.URL `json:"-"` // optional: override E.Host and LLM_EMBED_HOST env
}

EmbedRequest is the request passed to E.Embed.

func NewEmbed added in v0.2.2 

func NewEmbed(s string) EmbedRequest

NewEmbed request

func NewNomicDocument added in v0.2.2 

func NewNomicDocument(s string) EmbedRequest

NewNomicDocument adds `search_document: ` prefix to the given string

func NewNomicQuery added in v0.2.2 

func NewNomicQuery(s string) EmbedRequest

NewNomicQuery adds `search_query: ` prefix to the given string

type EmbedResponse added in v0.2.2 

type EmbedResponse struct {
	Model      string                `json:"model"`
	Embeddings []vector256.Vector256 `json:"embeddings"` // get resp.Embeddings[0]
	Data       []*struct {
		Embedding vector256.Vector256
	} `json:"data,omitempty"`
	Stats // limited for this endpoint
}

EmbedResponse is the response from [Client.Embed].

func (*EmbedResponse) GetEmbeddings added in v0.2.2 

func (e *EmbedResponse) GetEmbeddings() vector256.Vector256

type ErrorWithResponse added in v0.2.2 

type ErrorWithResponse struct {
	// contains filtered or unexported fields
}

func (ErrorWithResponse) Body added in v0.2.2 

func (e ErrorWithResponse) Body() []byte

func (ErrorWithResponse) Unwrap added in v0.2.2 

func (e ErrorWithResponse) Unwrap() error

Unwrap for errors.Unwrap.

type Function added in v0.3.2 

type Function[T any] struct {
	Name      string `json:"name"`
	Arguments T      `json:"arguments,omitempty"`
}

func (Function[T]) ArgsToJSON added in v0.3.2 

func (f Function[T]) ArgsToJSON() json.RawMessage

func (Function[T]) ArgsToMap added in v0.3.2 

func (f Function[T]) ArgsToMap() map[string]any

func (Function[T]) ToJSON added in v0.3.2 

func (f Function[T]) ToJSON() json.RawMessage

type LChoices added in v0.3.2 

type LChoices struct {
	FinishReason string `json:"finish_reason"`
	Index        int    `json:"index"`
	Delta        LDelta `json:"delta"`
}

type LDelta added in v0.3.2 

type LDelta struct{}

type LMessage added in v0.3.2 

type LMessage struct {
	Choices           []LChoices `json:"choices"`
	Created           int        `json:"created"`
	ID                string     `json:"id"`
	Model             string     `json:"model"`
	SystemFingerprint string     `json:"system_fingerprint"`
	Object            string     `json:"object"`
	Timings           LTimings   `json:"timings"`
}

type LTimings added in v0.3.2 

type LTimings struct {
	CacheN              int     `json:"cache_n"`
	PromptN             int     `json:"prompt_n"`
	PromptMs            float64 `json:"prompt_ms"`
	PromptPerTokenMs    float64 `json:"prompt_per_token_ms"`
	PromptPerSecond     float64 `json:"prompt_per_second"`
	PredictedN          int     `json:"predicted_n"`
	PredictedMs         float64 `json:"predicted_ms"`
	PredictedPerTokenMs float64 `json:"predicted_per_token_ms"`
	PredictedPerSecond  float64 `json:"predicted_per_second"`
}

type Message 

type Message struct {
	Role       string            `json:"role,omitempty"`
	Content    string            `json:"content,omitempty"`
	Reasoning  string            `json:"thinking,omitempty" gorm:"default:null"`
	ToolCalls  []json.RawMessage `json:"tool_calls,omitempty" gorm:"type:jsonb;serializer:json;default:null;"`
	ToolName   string            `json:"tool_name,omitempty" gorm:"default:null"`
	ToolCallID string            `json:"tool_call_id,omitempty" gorm:"default:null"`
	Images     [][]byte          `json:"images,omitempty" gorm:"serializer:json;type:jsonb;default:null;"`
	Message2   `gorm:"-"`
}

Message from LMessage

func ReplyTool added in v0.3.2 

func ReplyTool[T any](call Call[T], val any) Message

func ReplyToolOrError added in v0.3.2 

func ReplyToolOrError[T any](call Call[T], val any, err error) Message

func (Message) Add 

func (m Message) Add(m2 Message) Message

Add fields of c2 to c for gathering streamed chunks

Used in ChatResponse.Handler

func (Message) Equal 

func (m Message) Equal(m2 Message) bool

func (Message) IsEmpty 

func (m Message) IsEmpty() bool

func (Message) ToChatMessage 

func (m Message) ToChatMessage(from string) ChatMessage

ToChatMessage with DefaultThread, new MessageID, new timestamp.

Should be called ONCE immediately after stream is finished.

func (Message) ToThreadMessage added in v0.3.2 

func (m Message) ToThreadMessage(from string, thread uuid.UUID) ChatMessage

type Message2 added in v0.3.2 

type Message2 struct {
	// DO NOT USE
	// ToolCalls2  *Call[json.RawMessage] `json:"tool_call,omitempty" gorm:"-"`         // Internal, DO NOT USE
	ToolCallID2 string `json:"id,omitempty" gorm:"-"`                // Internal, DO NOT USE
	Reasoning2  string `json:"reasoning,omitempty" gorm:"-"`         // Internal, DO NOT USE
	Reasoning3  string `json:"reasoning_content,omitempty" gorm:"-"` // Internal, DO NOT USE
}

type Messages 

type Messages []Message

func (Messages) Top added in v0.2.5 

func (m Messages) Top() Message

func (Messages) TopN added in v0.2.5 

func (m Messages) TopN(n int) Messages

func (Messages) TopRole added in v0.2.5 

func (m Messages) TopRole(role string) Message

type Name added in v0.3.2 

type Name struct {
	Name     string `json:"name,omitempty"` // "function" always?
	Function struct {
		Name string `json:"name,omitempty"`
	} `json:"function"`
}

func (Name) GetName added in v0.3.2 

func (n Name) GetName() string

type Options 

type Options struct {
	Seed             int64    `json:"seed,omitempty"`
	Temperature      *float64 `json:"temperature,omitempty"` // 0.0 to 2.0
	NumCtx           int      `json:"num_ctx,omitempty"`
	TopP             *float64 `json:"top_p,omitempty"`             // 0.0 to 1.0
	TopK             *int     `json:"top_k,omitempty"`             // 1 to ???
	FrequencyPenalty *float64 `json:"frequency_penalty,omitempty"` // -2.0 to 2.0
	PresencePenalty  *float64 `json:"presence_penalty,omitempty"`  // -2.0 to 2.0
	//
	Debug int `json:"-"` // internal, for printing full chatrequest before sending
}

Options for ChatRequest

type RawToolCall added in v0.3.1 

type RawToolCall = Call[json.RawMessage]

type Spec added in v0.3.2 

type Spec = json.RawMessage

type Stats 

type Stats struct {
	LoadDuration       time.Duration `json:"load_duration,omitempty"`
	PromptEvalDuration time.Duration `json:"prompt_eval_duration,omitempty"`
	EvalDuration       time.Duration `json:"eval_duration,omitempty"`
	TotalDuration      time.Duration `json:"total_duration,omitempty"`
	PromptEvalCount    int           `json:"prompt_eval_count,omitempty"`
	EvalCount          int           `json:"eval_count,omitempty"`
	Usage
}

Stats has [Stats.String]

func (Stats) Summary added in v0.2.1 

func (st Stats) Summary(w io.Writer) error

Summary in well-known format

func (Stats) SummaryString added in v0.2.1 

func (st Stats) SummaryString() string

SummaryString collects [ChatResponse.Stats] into a string

type StreamHandler 

type StreamHandler func(cr ChatResponse) error

func NewHandlerOut 

func NewHandlerOut(w io.Writer, thinkpre string, thinkpost string) StreamHandler

NewHandlerOut plain 🍔

func (StreamHandler) HandleChat 

func (c StreamHandler) HandleChat(cr ChatResponse) error

type StringCalls added in v0.3.2 

type StringCalls []Call[string]

func (StringCalls) Combined added in v0.3.2 

func (c StringCalls) Combined() Calls[string]

Combined fragments of tool calls into final calls (one per index) instead of bunch of fragments

type Think 

type Think struct {
	Value `json:"-"`
}

Think Value should be one of: nil,"low","medium","high",true,false

func (Think) MarshalJSON 

func (t Think) MarshalJSON() ([]byte, error)

MarshalJSON implements json.Marshaler for sending request

type Tool added in v0.3.1 

type Tool struct {
	Spec     // json.RawMessage     // Spec, contains name
	Exec any `json:"-"`
}

Tool has JSON spec and Exec func

Your toolrunner can parse args and pass to Exec how you want to.

func (Tool) GetName added in v0.3.2 

func (t Tool) GetName() string

func (Tool) MarshalJSON added in v0.3.1 

func (t Tool) MarshalJSON() ([]byte, error)

type Tools added in v0.3.2 

type Tools []Tool

func (Tools) ByName added in v0.3.2 

func (tt Tools) ByName(name string) Tool

func (Tools) Run added in v0.3.2 

func (tt Tools) Run(ctx context.Context, name string, args any) (output any, err error)

Run returns the raw tool output, such as `42` or `"foo"` or `{"foo": 42}`

Tool Runner should wrap this in a ReplyTool or ReplyToolOrError

type Usage added in v0.3.1 

type Usage struct {
	TotalTokens  int `json:"total_tokens,omitempty"`
	PromptTokens int `json:"prompt_tokens,omitempty"`
}

type Value added in v0.3.1 

type Value = any

type W added in v0.3.2 

type W interface{ New() StreamHandler }

type W1 added in v0.3.2 

type W1 struct {
	// Writer if non-nil will be automatically added as a stream handler to [[E.Get]] and [[E.Stream]]
	Writer    io.Writer `json:"-"`
	ThinkPre  string    // optional, default used in [NewEndpoint]
	ThinkPost string    // optional, default used in [NewEndpoint]
}

func (W1) New added in v0.3.2 

func (w W1) New() StreamHandler

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.