Documentation
¶
Overview ¶
This file adapts upstream packages/llm/llm/src/assembler.ts to the simplified text/reasoning/tool-call/done protocol. While the agent logs raw chunks, BlockAssembler is the authoritative chunk-to-message conversion: adjacent text or reasoning deltas merge, tool calls become blocks on arrival, and output preserves stream order.
This file implements DeepSeek prefix-cache instrumentation (N01), following the API's KV-cache usage fields. CacheStats aggregates hit/miss tokens; ComputeHitRatio handles empty, all-hit, and all-miss usage without NaN. CacheLogLine provides cache.hit_ratio for monitoring, N08 alerts, and the cache acceptance-test baseline.
This file implements the R05 errorChain parity fix, following upstream packages/llm/llm/src/error.ts. RenderErrorChain accepts errors or values with a message field and renders the full cause chain. Identical wrapper/cause messages are not repeated. Single-cause Unwrap and aggregate Unwrap []error are supported; aggregate members render as [a; b]. An active-path set collapses true cycles to <circular cause> while fully expanding shared causes.
Package llm defines provider integration and streaming protocol types, following upstream packages/llm/llm and providers. LLMAdapter supports streaming Chat with usage reporting. Messages contain text, tool-use, tool-result, or image blocks; streams carry text, reasoning, tool calls, and completion. LlmFailure provides stable provider-error categories for S15 retries and the M34 request-error waterfall, matching upstream error.ts.
This file implements S15 LLM retries, following upstream retry middleware. RetryMiddleware retries overload and rate-limit failures with exponential backoff and jitter; refusal and context overflow propagate immediately. Each retry records attempt, backoffMs, and error in llm/retry. Exhausting attempts returns the last error for the M34 request-error waterfall.
Index ¶
- func CacheLogLine(u Usage) string
- func HitRatioOf(u Usage) float64
- func RenderErrorChain(v any) string
- type Alert
- type AlertLevel
- type AlertSink
- type BlockAssembler
- type CacheAlert
- type CacheStats
- type ChatRequest
- type ContentBlock
- type ContentBlockKind
- type ImageContent
- type LLMAdapter
- type LlmFailure
- type LlmFailureKind
- type Message
- type RetryConfig
- type RetryMiddleware
- type RetryWriter
- type Role
- type StreamChunk
- type StreamChunkKind
- type ToolCall
- type ToolResultContent
- type ToolSchema
- type Usage
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func CacheLogLine ¶
CacheLogLine formats a request log field such as cache.hit_ratio=0.95 for N01 instrumentation and monitoring.
func HitRatioOf ¶
HitRatioOf computes the cache hit ratio directly from Usage.
func RenderErrorChain ¶
RenderErrorChain renders arbitrary values with their full cause chain. Errors use Error(), falling back to the type name if empty. Other values prefer a Message field, then ordinary formatting. Aggregate members append as [a; b]; a single cause appends with a colon unless identical to its wrapper. Use this only for diagnostics, never routing; route on stable Kind/code values.
Types ¶
type Alert ¶
type Alert struct {
// Level identifies severity.
Level AlertLevel `json:"level"`
// Message includes session ID and current/previous ratios.
Message string `json:"message"`
// Webhook requests notification after consecutive failures.
Webhook bool `json:"webhook,omitempty"`
}
Alert describes a cache hit-rate degradation alert.
type AlertLevel ¶
type AlertLevel string
AlertLevel identifies alert severity.
const ( AlertWarn AlertLevel = "warn" AlertError AlertLevel = "error" )
Alert severity levels.
type AlertSink ¶
type AlertSink func(a Alert)
AlertSink delivers alerts to logging or webhook handlers.
type BlockAssembler ¶
type BlockAssembler struct {
// contains filtered or unexported fields
}
BlockAssembler incrementally converts StreamChunk values to ContentBlock values. Use NewBlockAssembler rather than the zero value.
func NewBlockAssembler ¶
func NewBlockAssembler() *BlockAssembler
NewBlockAssembler creates an empty assembler.
func (*BlockAssembler) Blocks ¶
func (a *BlockAssembler) Blocks() []ContentBlock
Blocks returns a copy of assembled blocks in stream order, including any open buffer.
func (*BlockAssembler) Finished ¶
func (a *BlockAssembler) Finished() bool
Finished reports whether a done chunk has arrived.
func (*BlockAssembler) Message ¶
func (a *BlockAssembler) Message() Message
Message constructs an assistant message from assembled blocks.
func (*BlockAssembler) Push ¶
func (a *BlockAssembler) Push(c StreamChunk)
Push consumes a chunk in stream order.
type CacheAlert ¶
type CacheAlert struct {
// Threshold is the low hit-rate threshold; the default is 0.5.
Threshold float64
// DropWarn is the sudden-drop threshold relative to the previous ratio; default 0.3.
DropWarn float64
// ConsecutiveFails is the number of low ratios before an error; default five.
ConsecutiveFails int
// Sink receives alerts.
Sink AlertSink
// contains filtered or unexported fields
}
CacheAlert detects and classifies cache hit-rate degradation. A drop larger than DropWarn emits a warning with session/current/previous values. ConsecutiveFails observations below Threshold emit an error and optional webhook notification. Recovery to Threshold resets the consecutive count.
func NewCacheAlert ¶
func NewCacheAlert(sink AlertSink) *CacheAlert
NewCacheAlert creates an alert detector with default settings.
func (*CacheAlert) Alerts ¶
func (a *CacheAlert) Alerts() []Alert
Alerts returns a snapshot of accumulated alerts.
func (*CacheAlert) Observe ¶
func (a *CacheAlert) Observe(sessionID string, ratio float64)
Observe records a ratio and emits any alerts, using sessionID as context.
type CacheStats ¶
type CacheStats struct {
// HitTokens counts prompt tokens served from cache.
HitTokens int `json:"hitTokens"`
// MissTokens counts prompt tokens not served from cache.
MissTokens int `json:"missTokens"`
}
CacheStats aggregates prefix-cache usage for one or more requests.
func (*CacheStats) ComputeHitRatio ¶
func (c *CacheStats) ComputeHitRatio() float64
ComputeHitRatio returns hit/total, or zero for empty usage. All hits yield one and all misses yield zero; the result is never 0/0.
func (*CacheStats) RecordUsage ¶
func (c *CacheStats) RecordUsage(u Usage)
RecordCacheUsage adds usage to cumulative statistics for metering and monitoring.
func (*CacheStats) Total ¶
func (c *CacheStats) Total() int
Total returns hit plus miss prompt tokens.
type ChatRequest ¶
type ChatRequest struct {
// ReasoningEffort is an explicit, provider-owned reasoning setting.
ReasoningEffort string `json:"reasoningEffort,omitempty"`
// Model names the requested model, such as deepseek-chat.
Model string `json:"model"`
// Messages contains the conversation.
Messages []Message `json:"messages"`
// System holds the system prompt separately for provider assembly.
System string `json:"system,omitempty"`
// Tools contains available tool definitions.
Tools []ToolSchema `json:"tools,omitempty"`
// Temperature controls sampling.
Temperature float64 `json:"temperature,omitempty"`
// MaxTokens limits generated output.
MaxTokens int `json:"maxTokens,omitempty"`
}
ChatRequest describes one model conversation request.
type ContentBlock ¶
type ContentBlock struct {
Kind ContentBlockKind `json:"kind"`
Text string `json:"text,omitempty"`
ToolCall *ToolCall `json:"toolCall,omitempty"`
ToolResult *ToolResultContent `json:"toolResult,omitempty"`
Image *ImageContent `json:"image,omitempty"`
}
ContentBlock is a tagged content-block union within a message.
func ToolResult ¶
func ToolResult(tr *ToolResultContent) ContentBlock
ToolResult constructs a tool result block.
type ContentBlockKind ¶
type ContentBlockKind string
ContentBlockKind identifies a content block type.
const ( BlockText ContentBlockKind = "text" BlockReasoning ContentBlockKind = "reasoning" BlockToolUse ContentBlockKind = "tool_use" BlockToolResult ContentBlockKind = "tool_result" BlockImage ContentBlockKind = "image" )
Supported content block kinds.
type ImageContent ¶
type ImageContent struct {
URL string `json:"url,omitempty"`
Attachment string `json:"attachment,omitempty"`
}
ImageContent refers to an image URL or attachment.
type LLMAdapter ¶
type LLMAdapter interface {
// Name returns the adapter name, such as deepseek.
Name() string
// Chat streams chunks through cb and returns final usage.
Chat(ctx context.Context, req ChatRequest, cb func(StreamChunk)) (Usage, error)
}
LLMAdapter is the common model invocation interface.
type LlmFailure ¶
type LlmFailure struct {
Kind LlmFailureKind `json:"kind"`
Message string `json:"message"`
// Cause retains the underlying error, such as context.DeadlineExceeded or
// net.OpError, for errors.Is/As diagnostics. JSON omits it to avoid cycles and
// sensitive details. Providers preserve cancellation/network causes so
// errors.Is(f, context.Canceled) distinguishes cancellation from network faults.
Cause error `json:"-"`
}
LlmFailure represents a model request failure with a stable category.
func ClassifyLlmError ¶
func ClassifyLlmError(err error) *LlmFailure
ClassifyLlmError classifies arbitrary errors, using unknown when no category matches.
func NewProviderFailure ¶
func NewProviderFailure(detail string) *LlmFailure
NewProviderFailure classifies provider detail, falling back to unknown, and preserves the original detail in Message for diagnostics.
func (*LlmFailure) Error ¶
func (e *LlmFailure) Error() string
Error implements the error interface.
func (*LlmFailure) Unwrap ¶
func (e *LlmFailure) Unwrap() error
Unwrap exposes Cause to errors.Is and errors.As.
type LlmFailureKind ¶
type LlmFailureKind string
LlmFailureKind provides stable model-failure categories.
const ( // FailOverload indicates service overload; retrying is recommended. FailOverload LlmFailureKind = "overload" // FailRateLimit indicates rate limiting; retry with backoff. FailRateLimit LlmFailureKind = "rate-limit" // FailResponseRefusal indicates a model refusal and is not retryable. FailResponseRefusal LlmFailureKind = "response-refusal" // FailContextOverflow indicates the context requires compaction. FailContextOverflow LlmFailureKind = "context-overflow" // FailQuota indicates exhausted quota or balance (QUOTA); default retries exclude it. FailQuota LlmFailureKind = "quota" // FailEmptyResponse indicates normal completion without content (EMPTY_RESPONSE). // It yields no assistant message and can be retried safely. FailEmptyResponse LlmFailureKind = "empty-response" // FailInvalidCredential indicates malformed credentials (INVALID_CREDENTIAL), // not missing credentials: replace the stored value rather than add one. // Default retries exclude this category. FailInvalidCredential LlmFailureKind = "invalid-credential" )
Failure categories correspond to stable upstream HarnessError codes in packages/llm/llm/src/error.ts, including CONTEXT_WINDOW_EXCEEDED, QUOTA, EMPTY_RESPONSE, INVALID_CREDENTIAL, RATE_LIMIT, and overload. Always route by Kind; never parse human-readable error messages.
func ClassifyProviderDetail ¶
func ClassifyProviderDetail(detail string) LlmFailureKind
ClassifyProviderDetail normalizes combined provider code/type/message details and maps them to a stable LlmFailureKind. Unrecognized details return empty. Patterns follow upstream context-window and quota classifiers.
func (LlmFailureKind) Retryable ¶
func (f LlmFailureKind) Retryable() bool
Retryable matches the upstream default set: overload, rate-limit, context-overflow after compaction, and empty-response. Refusal, quota, invalid credentials, and unknown categories are not retryable.
type Message ¶
type Message struct {
Role Role `json:"role"`
Content []ContentBlock `json:"content"`
}
Message combines a role and a list of content blocks.
func NewAssistantText ¶
NewAssistantText constructs an assistant text message.
func NewUserMessage ¶
NewUserMessage constructs a user text message.
type RetryConfig ¶
type RetryConfig struct {
// MaxAttempts includes the initial call; nonpositive values default to three.
MaxAttempts int
// BaseBackoff is the initial backoff duration.
BaseBackoff time.Duration
// MaxBackoff caps the backoff duration.
MaxBackoff time.Duration
// Jitter is a fraction from zero to one; zero gives deterministic backoff.
Jitter float64
}
RetryConfig configures retry behavior.
func DefaultRetryConfig ¶
func DefaultRetryConfig() RetryConfig
DefaultRetryConfig returns the default retry settings.
type RetryMiddleware ¶
type RetryMiddleware struct {
// contains filtered or unexported fields
}
RetryMiddleware decorates an LLM adapter with retries.
func NewRetryMiddleware ¶
func NewRetryMiddleware(inner LLMAdapter, cfg RetryConfig, write RetryWriter) *RetryMiddleware
NewRetryMiddleware creates a retry decorator; write can record each retry event.
func (*RetryMiddleware) Chat ¶
func (m *RetryMiddleware) Chat(ctx context.Context, req ChatRequest, cb func(StreamChunk)) (Usage, error)
Chat performs a streaming conversation with retries.
func (*RetryMiddleware) Name ¶
func (m *RetryMiddleware) Name() string
Name forwards the wrapped adapter name.
type RetryWriter ¶
type RetryWriter func(data session.LLMRetryData)
RetryWriter records llm/retry events, typically through SessionLog.Append.
type StreamChunk ¶
type StreamChunk struct {
Kind StreamChunkKind `json:"kind"`
Text string `json:"text,omitempty"`
Reasoning string `json:"reasoning,omitempty"`
ToolCall *ToolCall `json:"toolCall,omitempty"`
}
StreamChunk is one streaming callback payload.
type StreamChunkKind ¶
type StreamChunkKind string
StreamChunkKind identifies a streaming chunk type.
const ( ChunkText StreamChunkKind = "text" ChunkReasoning StreamChunkKind = "reasoning" ChunkToolCall StreamChunkKind = "tool-call" ChunkDone StreamChunkKind = "done" )
Supported streaming chunk kinds.
type ToolCall ¶
type ToolCall struct {
ID string `json:"id"`
Name string `json:"name"`
Input map[string]any `json:"input"`
}
ToolCall describes a model-requested tool invocation.
type ToolResultContent ¶
type ToolResultContent struct {
ToolUseID string `json:"toolUseId"`
Content string `json:"content"`
IsError bool `json:"isError,omitempty"`
}
ToolResultContent contains a tool result block.
type ToolSchema ¶
type ToolSchema struct {
Name string `json:"name"`
Description string `json:"description"`
Parameters map[string]any `json:"parameters"`
}
ToolSchema describes a model-visible tool using OpenAI function conventions.
type Usage ¶
type Usage struct {
PromptTokens int `json:"promptTokens"`
CompletionTokens int `json:"completionTokens"`
PromptCacheHitTokens int `json:"promptCacheHitTokens,omitempty"`
PromptCacheMissTokens int `json:"promptCacheMissTokens,omitempty"`
}
Usage records request token usage, including cache statistics for instrumentation.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package provider_deepseek implements the DeepSeek REST and SSE adapter, following upstream packages/llm/providers.
|
Package provider_deepseek implements the DeepSeek REST and SSE adapter, following upstream packages/llm/providers. |