llm

package
v0.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 9, 2026 License: MIT Imports: 10 Imported by: 0

Documentation

Overview

This file adapts upstream packages/llm/llm/src/assembler.ts to the simplified text/reasoning/tool-call/done protocol. While the agent logs raw chunks, BlockAssembler is the authoritative chunk-to-message conversion: adjacent text or reasoning deltas merge, tool calls become blocks on arrival, and output preserves stream order.

This file implements DeepSeek prefix-cache instrumentation (N01), following the API's KV-cache usage fields. CacheStats aggregates hit/miss tokens; ComputeHitRatio handles empty, all-hit, and all-miss usage without NaN. CacheLogLine provides cache.hit_ratio for monitoring, N08 alerts, and the cache acceptance-test baseline.

This file implements the R05 errorChain parity fix, following upstream packages/llm/llm/src/error.ts. RenderErrorChain accepts errors or values with a message field and renders the full cause chain. Identical wrapper/cause messages are not repeated. Single-cause Unwrap and aggregate Unwrap []error are supported; aggregate members render as [a; b]. An active-path set collapses true cycles to <circular cause> while fully expanding shared causes.

Package llm defines provider integration and streaming protocol types, following upstream packages/llm/llm and providers. LLMAdapter supports streaming Chat with usage reporting. Messages contain text, tool-use, tool-result, or image blocks; streams carry text, reasoning, tool calls, and completion. LlmFailure provides stable provider-error categories for S15 retries and the M34 request-error waterfall, matching upstream error.ts.

This file implements S15 LLM retries, following upstream retry middleware. RetryMiddleware retries overload and rate-limit failures with exponential backoff and jitter; refusal and context overflow propagate immediately. Each retry records attempt, backoffMs, and error in llm/retry. Exhausting attempts returns the last error for the M34 request-error waterfall.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CacheLogLine

func CacheLogLine(u Usage) string

CacheLogLine formats a request log field such as cache.hit_ratio=0.95 for N01 instrumentation and monitoring.

func HitRatioOf

func HitRatioOf(u Usage) float64

HitRatioOf computes the cache hit ratio directly from Usage.

func RenderErrorChain

func RenderErrorChain(v any) string

RenderErrorChain renders arbitrary values with their full cause chain. Errors use Error(), falling back to the type name if empty. Other values prefer a Message field, then ordinary formatting. Aggregate members append as [a; b]; a single cause appends with a colon unless identical to its wrapper. Use this only for diagnostics, never routing; route on stable Kind/code values.

Types

type Alert

type Alert struct {
	// Level identifies severity.
	Level AlertLevel `json:"level"`
	// Message includes session ID and current/previous ratios.
	Message string `json:"message"`
	// Webhook requests notification after consecutive failures.
	Webhook bool `json:"webhook,omitempty"`
}

Alert describes a cache hit-rate degradation alert.

type AlertLevel

type AlertLevel string

AlertLevel identifies alert severity.

const (
	AlertWarn  AlertLevel = "warn"
	AlertError AlertLevel = "error"
)

Alert severity levels.

type AlertSink

type AlertSink func(a Alert)

AlertSink delivers alerts to logging or webhook handlers.

type BlockAssembler

type BlockAssembler struct {
	// contains filtered or unexported fields
}

BlockAssembler incrementally converts StreamChunk values to ContentBlock values. Use NewBlockAssembler rather than the zero value.

func NewBlockAssembler

func NewBlockAssembler() *BlockAssembler

NewBlockAssembler creates an empty assembler.

func (*BlockAssembler) Blocks

func (a *BlockAssembler) Blocks() []ContentBlock

Blocks returns a copy of assembled blocks in stream order, including any open buffer.

func (*BlockAssembler) Finished

func (a *BlockAssembler) Finished() bool

Finished reports whether a done chunk has arrived.

func (*BlockAssembler) Message

func (a *BlockAssembler) Message() Message

Message constructs an assistant message from assembled blocks.

func (*BlockAssembler) Push

func (a *BlockAssembler) Push(c StreamChunk)

Push consumes a chunk in stream order.

type CacheAlert

type CacheAlert struct {
	// Threshold is the low hit-rate threshold; the default is 0.5.
	Threshold float64
	// DropWarn is the sudden-drop threshold relative to the previous ratio; default 0.3.
	DropWarn float64
	// ConsecutiveFails is the number of low ratios before an error; default five.
	ConsecutiveFails int
	// Sink receives alerts.
	Sink AlertSink
	// contains filtered or unexported fields
}

CacheAlert detects and classifies cache hit-rate degradation. A drop larger than DropWarn emits a warning with session/current/previous values. ConsecutiveFails observations below Threshold emit an error and optional webhook notification. Recovery to Threshold resets the consecutive count.

func NewCacheAlert

func NewCacheAlert(sink AlertSink) *CacheAlert

NewCacheAlert creates an alert detector with default settings.

func (*CacheAlert) Alerts

func (a *CacheAlert) Alerts() []Alert

Alerts returns a snapshot of accumulated alerts.

func (*CacheAlert) Observe

func (a *CacheAlert) Observe(sessionID string, ratio float64)

Observe records a ratio and emits any alerts, using sessionID as context.

type CacheStats

type CacheStats struct {
	// HitTokens counts prompt tokens served from cache.
	HitTokens int `json:"hitTokens"`
	// MissTokens counts prompt tokens not served from cache.
	MissTokens int `json:"missTokens"`
}

CacheStats aggregates prefix-cache usage for one or more requests.

func (*CacheStats) Add

func (c *CacheStats) Add(hit, miss int)

Add accumulates one usage record.

func (*CacheStats) ComputeHitRatio

func (c *CacheStats) ComputeHitRatio() float64

ComputeHitRatio returns hit/total, or zero for empty usage. All hits yield one and all misses yield zero; the result is never 0/0.

func (*CacheStats) RecordUsage

func (c *CacheStats) RecordUsage(u Usage)

RecordCacheUsage adds usage to cumulative statistics for metering and monitoring.

func (*CacheStats) Total

func (c *CacheStats) Total() int

Total returns hit plus miss prompt tokens.

type ChatRequest

type ChatRequest struct {
	// ReasoningEffort is an explicit, provider-owned reasoning setting.
	ReasoningEffort string `json:"reasoningEffort,omitempty"`
	// Model names the requested model, such as deepseek-chat.
	Model string `json:"model"`
	// Messages contains the conversation.
	Messages []Message `json:"messages"`
	// System holds the system prompt separately for provider assembly.
	System string `json:"system,omitempty"`
	// Tools contains available tool definitions.
	Tools []ToolSchema `json:"tools,omitempty"`
	// Temperature controls sampling.
	Temperature float64 `json:"temperature,omitempty"`
	// MaxTokens limits generated output.
	MaxTokens int `json:"maxTokens,omitempty"`
}

ChatRequest describes one model conversation request.

type ContentBlock

type ContentBlock struct {
	Kind       ContentBlockKind   `json:"kind"`
	Text       string             `json:"text,omitempty"`
	ToolCall   *ToolCall          `json:"toolCall,omitempty"`
	ToolResult *ToolResultContent `json:"toolResult,omitempty"`
	Image      *ImageContent      `json:"image,omitempty"`
}

ContentBlock is a tagged content-block union within a message.

func Image

func Image(img *ImageContent) ContentBlock

Image constructs an image block.

func Reasoning

func Reasoning(s string) ContentBlock

Reasoning constructs a reasoning block.

func Text

func Text(s string) ContentBlock

Text constructs a text block.

func ToolResult

func ToolResult(tr *ToolResultContent) ContentBlock

ToolResult constructs a tool result block.

func ToolUse

func ToolUse(tc *ToolCall) ContentBlock

ToolUse constructs a tool invocation block.

type ContentBlockKind

type ContentBlockKind string

ContentBlockKind identifies a content block type.

const (
	BlockText       ContentBlockKind = "text"
	BlockReasoning  ContentBlockKind = "reasoning"
	BlockToolUse    ContentBlockKind = "tool_use"
	BlockToolResult ContentBlockKind = "tool_result"
	BlockImage      ContentBlockKind = "image"
)

Supported content block kinds.

type ImageContent

type ImageContent struct {
	URL        string `json:"url,omitempty"`
	Attachment string `json:"attachment,omitempty"`
}

ImageContent refers to an image URL or attachment.

type LLMAdapter

type LLMAdapter interface {
	// Name returns the adapter name, such as deepseek.
	Name() string
	// Chat streams chunks through cb and returns final usage.
	Chat(ctx context.Context, req ChatRequest, cb func(StreamChunk)) (Usage, error)
}

LLMAdapter is the common model invocation interface.

type LlmFailure

type LlmFailure struct {
	Kind    LlmFailureKind `json:"kind"`
	Message string         `json:"message"`
	// Cause retains the underlying error, such as context.DeadlineExceeded or
	// net.OpError, for errors.Is/As diagnostics. JSON omits it to avoid cycles and
	// sensitive details. Providers preserve cancellation/network causes so
	// errors.Is(f, context.Canceled) distinguishes cancellation from network faults.
	Cause error `json:"-"`
}

LlmFailure represents a model request failure with a stable category.

func ClassifyLlmError

func ClassifyLlmError(err error) *LlmFailure

ClassifyLlmError classifies arbitrary errors, using unknown when no category matches.

func NewProviderFailure

func NewProviderFailure(detail string) *LlmFailure

NewProviderFailure classifies provider detail, falling back to unknown, and preserves the original detail in Message for diagnostics.

func (*LlmFailure) Error

func (e *LlmFailure) Error() string

Error implements the error interface.

func (*LlmFailure) Unwrap

func (e *LlmFailure) Unwrap() error

Unwrap exposes Cause to errors.Is and errors.As.

type LlmFailureKind

type LlmFailureKind string

LlmFailureKind provides stable model-failure categories.

const (
	// FailOverload indicates service overload; retrying is recommended.
	FailOverload LlmFailureKind = "overload"
	// FailRateLimit indicates rate limiting; retry with backoff.
	FailRateLimit LlmFailureKind = "rate-limit"
	// FailResponseRefusal indicates a model refusal and is not retryable.
	FailResponseRefusal LlmFailureKind = "response-refusal"
	// FailContextOverflow indicates the context requires compaction.
	FailContextOverflow LlmFailureKind = "context-overflow"
	// FailQuota indicates exhausted quota or balance (QUOTA); default retries exclude it.
	FailQuota LlmFailureKind = "quota"
	// FailEmptyResponse indicates normal completion without content (EMPTY_RESPONSE).
	// It yields no assistant message and can be retried safely.
	FailEmptyResponse LlmFailureKind = "empty-response"
	// FailInvalidCredential indicates malformed credentials (INVALID_CREDENTIAL),
	// not missing credentials: replace the stored value rather than add one.
	// Default retries exclude this category.
	FailInvalidCredential LlmFailureKind = "invalid-credential"
)

Failure categories correspond to stable upstream HarnessError codes in packages/llm/llm/src/error.ts, including CONTEXT_WINDOW_EXCEEDED, QUOTA, EMPTY_RESPONSE, INVALID_CREDENTIAL, RATE_LIMIT, and overload. Always route by Kind; never parse human-readable error messages.

func ClassifyProviderDetail

func ClassifyProviderDetail(detail string) LlmFailureKind

ClassifyProviderDetail normalizes combined provider code/type/message details and maps them to a stable LlmFailureKind. Unrecognized details return empty. Patterns follow upstream context-window and quota classifiers.

func (LlmFailureKind) Retryable

func (f LlmFailureKind) Retryable() bool

Retryable matches the upstream default set: overload, rate-limit, context-overflow after compaction, and empty-response. Refusal, quota, invalid credentials, and unknown categories are not retryable.

type Message

type Message struct {
	Role    Role           `json:"role"`
	Content []ContentBlock `json:"content"`
}

Message combines a role and a list of content blocks.

func NewAssistantText

func NewAssistantText(text string) Message

NewAssistantText constructs an assistant text message.

func NewUserMessage

func NewUserMessage(text string) Message

NewUserMessage constructs a user text message.

type RetryConfig

type RetryConfig struct {
	// MaxAttempts includes the initial call; nonpositive values default to three.
	MaxAttempts int
	// BaseBackoff is the initial backoff duration.
	BaseBackoff time.Duration
	// MaxBackoff caps the backoff duration.
	MaxBackoff time.Duration
	// Jitter is a fraction from zero to one; zero gives deterministic backoff.
	Jitter float64
}

RetryConfig configures retry behavior.

func DefaultRetryConfig

func DefaultRetryConfig() RetryConfig

DefaultRetryConfig returns the default retry settings.

type RetryMiddleware

type RetryMiddleware struct {
	// contains filtered or unexported fields
}

RetryMiddleware decorates an LLM adapter with retries.

func NewRetryMiddleware

func NewRetryMiddleware(inner LLMAdapter, cfg RetryConfig, write RetryWriter) *RetryMiddleware

NewRetryMiddleware creates a retry decorator; write can record each retry event.

func (*RetryMiddleware) Chat

func (m *RetryMiddleware) Chat(ctx context.Context, req ChatRequest, cb func(StreamChunk)) (Usage, error)

Chat performs a streaming conversation with retries.

func (*RetryMiddleware) Name

func (m *RetryMiddleware) Name() string

Name forwards the wrapped adapter name.

type RetryWriter

type RetryWriter func(data session.LLMRetryData)

RetryWriter records llm/retry events, typically through SessionLog.Append.

type Role

type Role string

Role identifies a message role.

const (
	RoleSystem    Role = "system"
	RoleUser      Role = "user"
	RoleAssistant Role = "assistant"
)

Supported message roles.

type StreamChunk

type StreamChunk struct {
	Kind      StreamChunkKind `json:"kind"`
	Text      string          `json:"text,omitempty"`
	Reasoning string          `json:"reasoning,omitempty"`
	ToolCall  *ToolCall       `json:"toolCall,omitempty"`
}

StreamChunk is one streaming callback payload.

type StreamChunkKind

type StreamChunkKind string

StreamChunkKind identifies a streaming chunk type.

const (
	ChunkText      StreamChunkKind = "text"
	ChunkReasoning StreamChunkKind = "reasoning"
	ChunkToolCall  StreamChunkKind = "tool-call"
	ChunkDone      StreamChunkKind = "done"
)

Supported streaming chunk kinds.

type ToolCall

type ToolCall struct {
	ID    string         `json:"id"`
	Name  string         `json:"name"`
	Input map[string]any `json:"input"`
}

ToolCall describes a model-requested tool invocation.

type ToolResultContent

type ToolResultContent struct {
	ToolUseID string `json:"toolUseId"`
	Content   string `json:"content"`
	IsError   bool   `json:"isError,omitempty"`
}

ToolResultContent contains a tool result block.

type ToolSchema

type ToolSchema struct {
	Name        string         `json:"name"`
	Description string         `json:"description"`
	Parameters  map[string]any `json:"parameters"`
}

ToolSchema describes a model-visible tool using OpenAI function conventions.

type Usage

type Usage struct {
	PromptTokens          int `json:"promptTokens"`
	CompletionTokens      int `json:"completionTokens"`
	PromptCacheHitTokens  int `json:"promptCacheHitTokens,omitempty"`
	PromptCacheMissTokens int `json:"promptCacheMissTokens,omitempty"`
}

Usage records request token usage, including cache statistics for instrumentation.

Directories

Path Synopsis
Package provider_deepseek implements the DeepSeek REST and SSE adapter, following upstream packages/llm/providers.
Package provider_deepseek implements the DeepSeek REST and SSE adapter, following upstream packages/llm/providers.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL