llm

package
v0.4.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 26, 2026 License: AGPL-3.0 Imports: 20 Imported by: 0

Documentation

Overview

Package llm provides interfaces and implementations for LLM backends.

This package defines the LLMClient interface for interacting with language models (Anthropic Claude, OpenAI, local models, etc.) and provides streaming support for real-time token generation.

Architecture

The package follows the interface-first pattern:

  • LLMClient interface defines the contract
  • AnthropicClient implements for Claude models
  • Additional implementations can be added for other backends

Streaming

Streaming is implemented via callback pattern. The ChatStream method calls a callback for each token as it's generated, enabling real-time display in CLI and SSE endpoints.

Thread Safety

All implementations must be safe for concurrent use.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type AnthropicClient 

type AnthropicClient struct {
	// contains filtered or unexported fields
}

func NewAnthropicClient 

func NewAnthropicClient() (*AnthropicClient, error)

func (*AnthropicClient) Chat 

func (a *AnthropicClient) Chat(ctx context.Context, messages []datatypes.Message, params GenerationParams) (string, error)

Chat implements the LLMClient interface

func (*AnthropicClient) ChatStream 

func (a *AnthropicClient) ChatStream(
	ctx context.Context,
	messages []datatypes.Message,
	params GenerationParams,
	callback StreamCallback,
) error

ChatStream implements streaming chat for the LLMClient interface.

Description

Sends a chat request to Anthropic with streaming enabled, then reads the SSE response line-by-line and calls the callback for each token. Handles both regular text tokens and thinking tokens.

Inputs

  • ctx: Context for cancellation and timeout.
  • messages: Conversation history.
  • params: Generation parameters.
  • callback: Called for each streaming event.

Outputs

  • error: Non-nil on network failure, API error, or callback abort.

Examples 

err := client.ChatStream(ctx, messages, params, func(e StreamEvent) error {
    if e.Type == StreamEventToken {
        fmt.Print(e.Content)
    }
    return nil
})

Limitations

  • Requires valid Anthropic API key
  • Timeout applies to entire stream duration

Assumptions

  • Anthropic API is available
  • Network is stable for stream duration

func (*AnthropicClient) Generate 

func (a *AnthropicClient) Generate(ctx context.Context, prompt string, params GenerationParams) (string, error)

Generate implements the LLMClient interface

type DefaultStreamProcessor 

type DefaultStreamProcessor struct {
	// contains filtered or unexported fields
}

DefaultStreamProcessor implements StreamProcessor with configurable behavior.

Description

DefaultStreamProcessor handles chunk processing with support for: - Thinking token redaction (privacy) - Response length limits (safety) - Thinking length limits (safety) - Rate limiting (backpressure)

Fields

  • cfg: Stream configuration.
  • rateLimiter: Optional rate limiter for callback invocations.
  • tokenCount: Running count of content tokens.
  • responseLen: Running total of response characters.
  • thinkingLen: Running total of thinking characters.

Thread Safety

Not thread-safe. Use one instance per stream.

Examples 

processor := NewDefaultStreamProcessor(StreamConfig{RedactThinking: true}, nil)
done, err := processor.ProcessChunk(ctx, chunk, callback)

Limitations

  • Single use per stream

Assumptions

  • Chunks arrive in order

func NewDefaultStreamProcessor 

func NewDefaultStreamProcessor(cfg StreamConfig, rateLimiter *rate.Limiter) *DefaultStreamProcessor

NewDefaultStreamProcessor creates a new DefaultStreamProcessor.

Description

Creates a processor with the given configuration and optional rate limiter. If cfg.RateLimitPerSecond > 0 and rateLimiter is nil, creates one automatically.

Inputs

  • cfg: Stream configuration.
  • rateLimiter: Optional pre-configured rate limiter. If nil and rate limiting is configured, one will be created.

Outputs

  • *DefaultStreamProcessor: Configured processor ready for use.

Examples 

// Auto-create rate limiter from config
p := NewDefaultStreamProcessor(StreamConfig{RateLimitPerSecond: 50}, nil)

// Use custom rate limiter
limiter := rate.NewLimiter(100, 10)
p := NewDefaultStreamProcessor(StreamConfig{}, limiter)

Limitations

  • Rate limiter is shared if passed in; be careful with concurrent streams

Assumptions

  • cfg has reasonable values (non-negative limits)

func (*DefaultStreamProcessor) GetResponseLength 

func (p *DefaultStreamProcessor) GetResponseLength() int

GetResponseLength returns the total response characters processed.

Description

Returns the total number of characters from content tokens, after any truncation from length limits.

Outputs

  • int: Total response characters.

Examples 

length := processor.GetResponseLength()

Limitations

  • Does not include thinking content

Assumptions

  • Called after processing is complete or for progress reporting

func (*DefaultStreamProcessor) GetTokenCount 

func (p *DefaultStreamProcessor) GetTokenCount() int

GetTokenCount returns the number of content tokens processed.

Description

Returns the running count of content tokens (chunks with non-empty Message.Content) processed so far.

Outputs

  • int: Number of content tokens.

Examples 

count := processor.GetTokenCount()

Limitations

  • Does not count thinking tokens

Assumptions

  • Called after processing is complete or for progress reporting

func (*DefaultStreamProcessor) ProcessChunk 

func (p *DefaultStreamProcessor) ProcessChunk(ctx context.Context, chunk *ollamaStreamChunk, callback StreamCallback) (bool, error)

ProcessChunk processes a single chunk and emits appropriate events.

Description

Handles a single NDJSON chunk from Ollama streaming response: 1. Checks for error in chunk and emits error event 2. Processes thinking content (if present and not redacted) 3. Processes content tokens 4. Applies length limits and rate limiting 5. Returns done status from chunk

Inputs

  • ctx: Context for cancellation and rate limiter waiting.
  • chunk: Parsed Ollama stream chunk.
  • callback: Callback to invoke for each event.

Outputs

  • bool: True if chunk.Done is true (stream complete).
  • error: Non-nil on callback error, rate limiter error, or chunk error.

Examples 

chunk := &ollamaStreamChunk{Message: datatypes.Message{Content: "Hello"}}
done, err := processor.ProcessChunk(ctx, chunk, callback)

Limitations

  • Truncates content silently when limits exceeded
  • Logging should be added for truncation events

Assumptions

  • chunk is non-nil and valid
  • callback handles events quickly

type GenerationParams 

type GenerationParams struct {
	Temperature     *float32      `json:"temperature"`
	TopK            *int          `json:"top_k"`
	TopP            *float32      `json:"top_p"`
	MaxTokens       *int          `json:"max_tokens"`
	Stop            []string      `json:"stop"`
	ToolDefinitions []interface{} `json:"tools,omitempty"`
	EnableThinking  bool          `json:"thinking,omitempty"`
	BudgetTokens    int           `json:"budget_tokens,omitempty"`
}

GenerationParams holds parameters for LLM generation.

Description

Contains all configurable parameters for text generation including temperature, sampling parameters, and tool definitions. These parameters control the LLM's output behavior.

Fields

  • Temperature: Sampling temperature (0.0-1.0). Lower = more deterministic. nil uses the model's default.
  • TopK: Sample from top K tokens. nil uses model default.
  • TopP: Nucleus sampling threshold. nil uses model default.
  • MaxTokens: Maximum tokens to generate. nil uses model default.
  • Stop: Stop sequences to halt generation. Empty means no custom stops.
  • ToolDefinitions: Tool schemas for function calling (Claude tools format).
  • EnableThinking: Enable Claude extended thinking mode. Only works with Claude models that support extended thinking.
  • BudgetTokens: Token budget for thinking (max 65536). Only used when EnableThinking is true.

Examples 

// Default parameters
params := GenerationParams{}

// Custom temperature
temp := float32(0.7)
params := GenerationParams{Temperature: &temp}

// Extended thinking
params := GenerationParams{EnableThinking: true, BudgetTokens: 4096}

Limitations

  • Not all parameters are supported by all backends
  • EnableThinking only works with Claude models

Assumptions

  • nil values mean "use model default"

type HFTransformersClient 

type HFTransformersClient struct {
	// contains filtered or unexported fields
}

func (*HFTransformersClient) Chat 

func (h *HFTransformersClient) Chat(ctx context.Context, messages []datatypes.Message,
	params GenerationParams) (string, error)

Chat conducts a conversation with message history.

Description

Currently not implemented for HFTransformersClient. Returns an error indicating that this backend is not implemented.

Inputs

  • ctx: Context for cancellation and timeout.
  • messages: Conversation history.
  • params: Generation parameters.

Outputs

  • string: Empty string.
  • error: Always returns not implemented error.

Limitations

  • Not implemented.

Assumptions

  • None.

func (*HFTransformersClient) ChatStream 

func (h *HFTransformersClient) ChatStream(ctx context.Context, messages []datatypes.Message,
	params GenerationParams, callback StreamCallback) error

ChatStream streams a conversation response token-by-token.

Description

Currently not implemented for HFTransformersClient. Returns an error indicating that streaming is not supported for this backend.

Inputs

  • ctx: Context for cancellation and timeout.
  • messages: Conversation history.
  • params: Generation parameters.
  • callback: Callback for streaming events.

Outputs

  • error: Always returns ErrStreamingNotSupported.

Limitations

  • Streaming is not implemented for HuggingFace Transformers backend.

Assumptions

  • None.

func (*HFTransformersClient) Generate 

func (h *HFTransformersClient) Generate(ctx context.Context, prompt string,
	params GenerationParams) (string, error)

Generate produces text from a single prompt.

Description

Currently not implemented for HFTransformersClient. Returns an error indicating that this backend is not implemented.

Inputs

  • ctx: Context for cancellation and timeout.
  • prompt: Text prompt to complete.
  • params: Generation parameters.

Outputs

  • string: Empty string.
  • error: Always returns not implemented error.

Limitations

  • Not implemented.

Assumptions

  • None.

type LLMClient 

type LLMClient interface {
	// Generate produces text from a single prompt.
	//
	// # Description
	//
	// Sends a prompt to the LLM and returns the generated text.
	// This is a simple completion API without conversation context.
	// Prefer Chat for conversational interactions.
	//
	// # Inputs
	//
	//   - ctx: Context for cancellation and timeout. When cancelled,
	//     the method returns with context.Canceled error.
	//   - prompt: Text prompt to complete. Must not be empty.
	//   - params: Generation parameters. Use empty struct for defaults.
	//
	// # Outputs
	//
	//   - string: Generated text response.
	//   - error: Non-nil on network failure, API error, or cancellation.
	//
	// # Examples
	//
	//	response, err := client.Generate(ctx, "Explain OAuth in one sentence", GenerationParams{})
	//
	// # Limitations
	//
	//   - No conversation context (stateless)
	//   - Some backends may not support this method
	//
	// # Assumptions
	//
	//   - Prompt is within model's context window
	Generate(ctx context.Context, prompt string, params GenerationParams) (string, error)

	// Chat conducts a conversation with message history.
	//
	// # Description
	//
	// Sends a conversation (system, user, assistant messages) to the LLM
	// and returns the assistant's response. This is a blocking call that
	// waits for the complete response.
	//
	// # Inputs
	//
	//   - ctx: Context for cancellation and timeout.
	//   - messages: Conversation history. Must have at least one message.
	//     Messages should alternate user/assistant with optional system.
	//   - params: Generation parameters.
	//
	// # Outputs
	//
	//   - string: Assistant's complete response.
	//   - error: Non-nil on failure.
	//
	// # Examples
	//
	//	messages := []datatypes.Message{
	//	    {Role: "system", Content: "You are helpful."},
	//	    {Role: "user", Content: "What is 2+2?"},
	//	}
	//	response, err := client.Chat(ctx, messages, GenerationParams{})
	//
	// # Limitations
	//
	//   - Blocks until complete response received
	//   - No partial results on timeout
	//
	// # Assumptions
	//
	//   - Messages are well-formed with valid roles
	//   - Total tokens (input + output) within context window
	Chat(ctx context.Context, messages []datatypes.Message, params GenerationParams) (string, error)

	// ChatStream conducts a conversation with streaming response.
	//
	// # Description
	//
	// Like Chat, but streams the response token-by-token via callback.
	// Enables real-time display of generation progress. The callback
	// is called for each token as it's generated.
	//
	// If an error occurs during streaming, the callback is called with
	// a StreamEventError event before the method returns. This allows
	// callers to see partial results before the error.
	//
	// # Inputs
	//
	//   - ctx: Context for cancellation and timeout. Cancellation stops
	//     streaming and returns context.Canceled.
	//   - messages: Conversation history.
	//   - params: Generation parameters.
	//   - callback: Called for each streaming event. Return error to abort.
	//
	// # Outputs
	//
	//   - error: Non-nil on failure or if callback returns error.
	//     If streaming error occurs, callback receives error event first.
	//
	// # Examples
	//
	//	var fullResponse strings.Builder
	//	err := client.ChatStream(ctx, messages, params, func(e StreamEvent) error {
	//	    switch e.Type {
	//	    case StreamEventToken:
	//	        fullResponse.WriteString(e.Content)
	//	        fmt.Print(e.Content)  // Real-time display
	//	    case StreamEventThinking:
	//	        fmt.Printf("[thinking] %s", e.Content)
	//	    case StreamEventError:
	//	        fmt.Printf("[error] %s", e.Error)
	//	    }
	//	    return nil
	//	})
	//
	// # Limitations
	//
	//   - Callback errors abort the stream
	//   - No automatic retry on transient failures
	//
	// # Assumptions
	//
	//   - Backend supports streaming API
	//   - Callback handles events quickly
	ChatStream(ctx context.Context, messages []datatypes.Message, params GenerationParams, callback StreamCallback) error
}

LLMClient defines the standard interface for any LLM backend.

Description

LLMClient abstracts LLM interactions, enabling different backends (Anthropic Claude, OpenAI, local models) to be used interchangeably. The interface provides both blocking and streaming methods.

Methods

  • Generate: Single-prompt completion (legacy, prefer Chat)
  • Chat: Blocking conversation with full response
  • ChatStream: Streaming conversation with token-by-token callbacks

Thread Safety

Implementations must be safe for concurrent use. Multiple goroutines may call methods simultaneously.

Limitations

  • Not all backends support all features (e.g., extended thinking)
  • Streaming requires backend support

Assumptions

  • Backend is configured and authenticated before use
  • Context cancellation is respected

type LocalLlamaCppClient 

type LocalLlamaCppClient struct {
	// contains filtered or unexported fields
}

func NewLocalLlamaCppClient 

func NewLocalLlamaCppClient() (*LocalLlamaCppClient, error)

func (*LocalLlamaCppClient) Chat 

func (l *LocalLlamaCppClient) Chat(ctx context.Context, messages []datatypes.Message,
	params GenerationParams) (string, error)

Chat TODO: Implement

func (*LocalLlamaCppClient) ChatStream 

func (l *LocalLlamaCppClient) ChatStream(ctx context.Context, messages []datatypes.Message,
	params GenerationParams, callback StreamCallback) error

ChatStream streams a conversation response token-by-token.

Description

Currently not implemented for LocalLlamaCppClient. Returns an error indicating that streaming is not supported for this backend.

Inputs

  • ctx: Context for cancellation and timeout.
  • messages: Conversation history.
  • params: Generation parameters.
  • callback: Callback for streaming events.

Outputs

  • error: Always returns ErrStreamingNotSupported.

Limitations

  • Streaming is not implemented for llama.cpp backend.

Assumptions

  • None.

func (*LocalLlamaCppClient) Generate 

func (l *LocalLlamaCppClient) Generate(ctx context.Context, prompt string,
	params GenerationParams) (string, error)

Generate implements the LLMClient interface

type LocalLlamaCppClientPayload 

type LocalLlamaCppClientPayload struct {
	Prompt      string   `json:"prompt"`
	NPredict    int      `json:"n_predict"`
	Temperature *float32 `json:"temperature,omitempty"`
	TopK        *int     `json:"top_k,omitempty"`
	TopP        *float32 `json:"top_p,omitempty"`
	MaxTokens   *int     `json:"max_tokens,omitempty"`
	Stop        []string `json:"stop,omitempty"`
}

type OllamaClient 

type OllamaClient struct {
	// contains filtered or unexported fields
}

func NewOllamaClient 

func NewOllamaClient() (*OllamaClient, error)

func (*OllamaClient) Chat 

func (o *OllamaClient) Chat(ctx context.Context, messages []datatypes.Message,
	params GenerationParams) (string, error)

func (*OllamaClient) ChatStream 

func (o *OllamaClient) ChatStream(ctx context.Context, messages []datatypes.Message,
	params GenerationParams, callback StreamCallback) error

ChatStream streams a conversation response token-by-token.

Description

Streams responses from Ollama's /api/chat endpoint with stream=true. Each token is delivered via the callback as a StreamEvent. Supports thinking models (gpt-oss, DeepSeek-R1) which emit thinking tokens. Uses default StreamConfig.

Inputs

  • ctx: Context for cancellation and timeout. Cancellation stops streaming.
  • messages: Conversation history with system, user, assistant messages.
  • params: Generation parameters (temperature, max_tokens, etc).
  • callback: Called for each token. Return error to abort streaming.

Outputs

  • error: Non-nil on network failure, API error, or callback error.

Examples 

var response strings.Builder
err := client.ChatStream(ctx, messages, params, func(e StreamEvent) error {
    switch e.Type {
    case StreamEventToken:
        response.WriteString(e.Content)
        fmt.Print(e.Content)
    case StreamEventThinking:
        fmt.Printf("[thinking] %s", e.Content)
    case StreamEventError:
        return fmt.Errorf("stream error: %s", e.Error)
    }
    return nil
})

Limitations

  • Requires Ollama server to support streaming
  • Thinking field only populated by thinking-capable models

Assumptions

  • Ollama server is running and accessible
  • Model supports chat API

func (*OllamaClient) ChatStreamWithConfig 

func (o *OllamaClient) ChatStreamWithConfig(ctx context.Context, messages []datatypes.Message,
	params GenerationParams, callback StreamCallback, cfg StreamConfig) error

ChatStreamWithConfig streams with explicit configuration.

Description

Like ChatStream but accepts a StreamConfig for fine-grained control over privacy (thinking redaction), rate limiting, and length limits. This is the primary implementation; ChatStream delegates to this.

Inputs

  • ctx: Context for cancellation and timeout.
  • messages: Conversation history.
  • params: Generation parameters.
  • callback: Streaming event callback.
  • cfg: Streaming configuration.

Outputs

  • error: Non-nil on failure.

Examples 

cfg := StreamConfig{RedactThinking: true, RateLimitPerSecond: 50}
err := client.ChatStreamWithConfig(ctx, messages, params, callback, cfg)

Limitations

  • Rate limiting adds latency
  • Truncation loses content

Assumptions

  • Config values are reasonable (not negative, etc)

func (*OllamaClient) Generate 

func (o *OllamaClient) Generate(ctx context.Context, prompt string,
	params GenerationParams) (string, error)

Generate implements the LLMClient interface

type OpenAIClient 

type OpenAIClient struct {
	// contains filtered or unexported fields
}

func NewOpenAIClient 

func NewOpenAIClient() (*OpenAIClient, error)

func (*OpenAIClient) Chat 

func (o *OpenAIClient) Chat(ctx context.Context, messages []datatypes.Message, params GenerationParams) (string, error)

Chat TODO: Implement

func (*OpenAIClient) ChatStream 

func (o *OpenAIClient) ChatStream(ctx context.Context, messages []datatypes.Message,
	params GenerationParams, callback StreamCallback) error

ChatStream streams a conversation response token-by-token.

Description

Currently not implemented for OpenAIClient. Returns an error indicating that streaming is not supported for this backend.

Inputs

  • ctx: Context for cancellation and timeout.
  • messages: Conversation history.
  • params: Generation parameters.
  • callback: Callback for streaming events.

Outputs

  • error: Always returns ErrStreamingNotSupported.

Limitations

  • Streaming is not implemented for OpenAI backend.

Assumptions

  • None.

func (*OpenAIClient) Generate 

func (o *OpenAIClient) Generate(ctx context.Context, prompt string, params GenerationParams) (string, error)

Generate implements the LLMClient interface

type StreamCallback 

type StreamCallback func(event StreamEvent) error

StreamCallback is called for each event during streaming.

Description

StreamCallback receives events as they are generated by the LLM. Return an error to abort streaming (e.g., on client disconnect). The callback should process events quickly to avoid backpressure.

Inputs

  • event: The streaming event (token, thinking, or error).

Outputs

  • error: Non-nil to abort streaming. The ChatStream method will return this error after cleanup.

Examples 

callback := func(event StreamEvent) error {
    switch event.Type {
    case StreamEventToken:
        fmt.Print(event.Content)
    case StreamEventThinking:
        fmt.Printf("[thinking] %s", event.Content)
    case StreamEventError:
        return fmt.Errorf("stream error: %s", event.Error)
    }
    return nil
}

Limitations

  • Must handle events quickly to avoid backpressure
  • Errors abort the stream immediately

Assumptions

  • Called in event order (tokens arrive in generation order)
  • Called from a single goroutine (no concurrent calls)

type StreamConfig 

type StreamConfig struct {
	RedactThinking     bool `json:"redact_thinking"`
	MaxThinkingLength  int  `json:"max_thinking_length"`
	RateLimitPerSecond int  `json:"rate_limit_per_second"`
	MaxResponseLength  int  `json:"max_response_length"`
}

StreamConfig configures streaming behavior for ChatStream.

Description

StreamConfig provides fine-grained control over streaming behavior including privacy controls for thinking tokens, rate limiting to prevent overwhelming clients, and maximum length limits for safety.

Fields

  • RedactThinking: If true, thinking tokens from models like gpt-oss are not emitted to the callback. The thinking still occurs server-side.
  • MaxThinkingLength: Maximum characters for thinking content per stream. 0 means unlimited. Truncates if exceeded.
  • RateLimitPerSecond: Maximum callback invocations per second. 0 disables.
  • MaxResponseLength: Maximum total response characters. 0 means unlimited.

Examples 

// Privacy-focused config
cfg := StreamConfig{RedactThinking: true, MaxThinkingLength: 1000}

// Rate-limited streaming
cfg := StreamConfig{RateLimitPerSecond: 100}

Limitations

  • Rate limiting adds latency to token delivery
  • Truncation loses data (use logging to preserve if needed)

Assumptions

  • Reasonable limits prevent DoS from runaway generation

func DefaultStreamConfig 

func DefaultStreamConfig() StreamConfig

DefaultStreamConfig returns a StreamConfig with safe defaults.

Description

Returns a configuration suitable for most use cases: - Thinking tokens are passed through (not redacted) - No rate limiting (full speed) - 100KB max response length for safety

Outputs

  • StreamConfig: Default configuration.

type StreamEvent 

type StreamEvent struct {
	Type    StreamEventType
	Content string
	Error   string
}

StreamEvent represents a single event during LLM streaming.

Description

StreamEvent is emitted by ChatStream for each token or event during generation. The Type field indicates what kind of event this is and which fields are populated.

Fields

  • Type: Event type (token, thinking, error).
  • Content: Token content. Populated for token and thinking events.
  • Error: Error message. Populated for error events.

Examples 

// Token event
StreamEvent{Type: StreamEventToken, Content: "Hello"}

// Thinking event
StreamEvent{Type: StreamEventThinking, Content: "Let me analyze..."}

// Error event
StreamEvent{Type: StreamEventError, Error: "Connection reset"}

Limitations

  • Only one of Content or Error is populated per event

Assumptions

  • Events are delivered in generation order

type StreamEventType 

type StreamEventType string

StreamEventType represents the type of streaming event.

Description

StreamEventType categorizes streaming events to allow handlers to process different event types appropriately. Token events contain generated text, thinking events contain reasoning, and error events signal problems during generation. 

const (
	// StreamEventToken indicates a content token event.
	// The Content field contains the generated text fragment.
	StreamEventToken StreamEventType = "token"

	// StreamEventThinking indicates a thinking/reasoning token event.
	// The Content field contains Claude's reasoning text.
	// Only emitted when EnableThinking is true.
	StreamEventThinking StreamEventType = "thinking"

	// StreamEventError indicates an error occurred during streaming.
	// The Error field contains the error message.
	// Streaming typically stops after an error event.
	StreamEventError StreamEventType = "error"
)

type StreamProcessor 

type StreamProcessor interface {
	// ProcessChunk processes a single parsed chunk and invokes callback.
	//
	// # Inputs
	//   - ctx: Context for cancellation.
	//   - chunk: Parsed stream chunk.
	//   - callback: Event callback.
	//
	// # Outputs
	//   - bool: True if stream is done.
	//   - error: Non-nil on callback error or processing failure.
	ProcessChunk(ctx context.Context, chunk *ollamaStreamChunk, callback StreamCallback) (bool, error)

	// GetTokenCount returns total tokens processed so far.
	GetTokenCount() int

	// GetResponseLength returns total response characters processed.
	GetResponseLength() int
}

StreamProcessor defines the contract for processing streaming responses.

Description

StreamProcessor abstracts the streaming response processing logic, allowing for different implementations (e.g., testing, rate-limited). Implementations must be safe for single-threaded use within a stream.

Methods

  • ProcessChunk: Process a single NDJSON chunk and emit events.
  • GetTokenCount: Return total tokens processed.
  • GetResponseLength: Return total response characters.

Thread Safety

Not required to be thread-safe; called from single goroutine per stream.

Limitations

  • Single stream per processor instance

Assumptions

  • Called sequentially for each chunk in order

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.