llmclient

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 7, 2026 License: Apache-2.0 Imports: 27 Imported by: 4

README

Scitrera LLM Client for Go

A provider-neutral Go client for OpenAI Chat Completions, OpenAI Responses, Anthropic Messages, Gemini GenerateContent, and Amazon Bedrock Converse backends, plus OpenAI, Gemini, Amazon Titan, and Cohere embedding backends. The client consumes and returns types from github.com/scitrera/go-llm/protocol.

The package has no gateway or agent-runtime dependency. Callers provide backend configuration, authentication, and an http.Client; gateway concerns such as tenant routing, circuit breakers, shared retry budgets, persistence, and model lifecycle remain outside this module.

Retries are disabled by default. Streaming uses a capacity-one event channel to preserve downstream backpressure and supports separate first-event and idle liveness deadlines. Configured retries apply to buffered calls; streams are never replayed by the library after an attempt begins.

Default Gemini endpoints select generateContent or streamGenerateContent?alt=sse; embedding calls select embedContent or batchEmbedContents from input cardinality. Default Bedrock endpoints select converse or converse-stream, safely escaping the model as one URI label. Bedrock streams use a bounded AWS event-stream decoder with prelude/message CRC validation. SigV4Authenticator signs the finalized URL, headers, and body using standard-library cryptography. Its AWSCredentialsProvider remains caller-owned, so applications can plug in static material, an AWS SDK chain, or workload identity without this module owning discovery, refresh, or storage. Backend.Path, Backend.StreamPath, and Backend.BatchPath may override defaults and use a {model} placeholder. Titan and Cohere embeddings use the Bedrock InvokeModel endpoint and the same caller-owned SigV4 authenticator.

Client.Embed returns the rank-aware embedding contract from the protocol package; one response item can contain a validated rank-1 vector or rank-2 matrix without conflating that shape with request batch cardinality.

Backend.BodyDefaults supplies bounded, top-level provider request defaults (the analogue of an SDK extra_body). A caller-provided field always wins; defaults are validated before any network request and are limited to 64 fields, 1 MiB per value, and 4 MiB total. Protocol-owned fields such as model, messages, input, tools, and stream are rejected. Gateways which retain ownership of raw proxying can call MergeBodyDefaults to reuse the exact same bounded, request-wins semantics without adopting this client's retry policy.

Licensed under the Apache License, Version 2.0.

Run correctness tests and the request-default microbenchmark with:

go test ./...
go test -run '^$' -bench . -benchmem ./...

Documentation

Overview

Package llmclient provides provider-neutral buffered and streaming HTTP execution without importing gateway or agent-runtime concerns.

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrFirstEventTimeout = errors.New("llm client first stream event timeout")
	ErrStreamIdleTimeout = errors.New("llm client stream idle timeout")
)

Functions

func MergeBodyDefaults 

func MergeBodyDefaults(body json.RawMessage, defaults map[string]json.RawMessage) (json.RawMessage, error)

MergeBodyDefaults shallow-merges bounded provider-specific defaults into an encoded request object. Fields already present in body always win. This helper is exported so gateways that retain ownership of raw HTTP proxying can share the same validation and merge semantics as Client.

Types

type AWSCredentials 

type AWSCredentials struct {
	AccessKeyID     string
	SecretAccessKey string
	SessionToken    string
	ExpiresAt       time.Time
}

AWSCredentials is the minimal material required for SigV4. The client never discovers, refreshes, persists, or logs this value.

type AWSCredentialsProvider 

type AWSCredentialsProvider interface {
	Retrieve(context.Context) (AWSCredentials, error)
}

AWSCredentialsProvider lets a host integrate static credentials, an AWS SDK chain, workload identity, or another refresh mechanism without coupling this module to that policy.

func StaticAWSCredentials 

func StaticAWSCredentials(credentials AWSCredentials) AWSCredentialsProvider

type AWSCredentialsProviderFunc 

type AWSCredentialsProviderFunc func(context.Context) (AWSCredentials, error)

func (AWSCredentialsProviderFunc) Retrieve 

func (f AWSCredentialsProviderFunc) Retrieve(ctx context.Context) (AWSCredentials, error)

type AuthFunc 

type AuthFunc func(context.Context, *http.Request) error

func (AuthFunc) Apply 

func (f AuthFunc) Apply(ctx context.Context, request *http.Request) error

type Authenticator 

type Authenticator interface {
	Apply(context.Context, *http.Request) error
}

func APIKeyHeader 

func APIKeyHeader(name, value string) Authenticator

func BearerToken 

func BearerToken(token string) Authenticator

type Backend 

type Backend struct {
	Name         string
	Format       llmprotocol.Format
	BaseURL      string
	Path         string
	StreamPath   string
	BatchPath    string
	Model        string
	Headers      map[string]string
	BodyDefaults map[string]json.RawMessage
	Auth         Authenticator
}

func (Backend) Validate 

func (b Backend) Validate() error

type CallMetadata 

type CallMetadata struct {
	BackendName         string
	Format              llmprotocol.Format
	Endpoint            string
	StatusCode          int
	Attempts            int
	StartedAt           time.Time
	Duration            time.Duration
	ResponseHeaders     http.Header
	RequestDiagnostics  []llmprotocol.Diagnostic
	ResponseDiagnostics []llmprotocol.Diagnostic
}

type CallResult 

type CallResult struct {
	Response llmprotocol.Response
	Metadata CallMetadata
}

type Client 

type Client struct {
	// contains filtered or unexported fields
}

func New 

func New(options Options) *Client

func (*Client) Call 

func (c *Client) Call(ctx context.Context, backend Backend, request llmprotocol.Request) (CallResult, error)

func (*Client) Embed 

func (c *Client) Embed(ctx context.Context, backend Backend, request llmprotocol.EmbeddingRequest) (EmbeddingCallResult, error)

Embed executes one buffered embeddings request. Batch cardinality is derived from request.Inputs; each response Data item may independently contain a rank-one or rank-two tensor.

func (*Client) Stream 

func (c *Client) Stream(ctx context.Context, backend Backend, request llmprotocol.Request) (*Stream, error)

type EmbeddingCallResult 

type EmbeddingCallResult struct {
	Response llmprotocol.EmbeddingResponse
	Metadata CallMetadata
}

type Error 

type Error struct {
	Kind       ErrorKind
	StatusCode int
	Type       string
	Code       string
	Message    string
	Retryable  bool
	Cause      error
	Body       json.RawMessage
}

func (*Error) Error 

func (e *Error) Error() string

func (*Error) Unwrap 

func (e *Error) Unwrap() error

type ErrorKind 

type ErrorKind string
const (
	ErrorInvalidConfig  ErrorKind = "invalid_config"
	ErrorAuthentication ErrorKind = "authentication"
	ErrorEncode         ErrorKind = "encode"
	ErrorTransport      ErrorKind = "transport"
	ErrorTimeout        ErrorKind = "timeout"
	ErrorCanceled       ErrorKind = "canceled"
	ErrorUpstream       ErrorKind = "upstream"
	ErrorResponseLimit  ErrorKind = "response_limit"
	ErrorDecode         ErrorKind = "decode"
	ErrorStream         ErrorKind = "stream"
)

type Options 

type Options struct {
	HTTPClient         *http.Client
	Registry           *codec.Registry
	EmbeddingRegistry  *codec.EmbeddingRegistry
	Policy             llmprotocol.Policy
	MaxResponseBytes   int64
	FirstEventTimeout  time.Duration
	StreamIdleTimeout  time.Duration
	MaxStreamLineBytes int
	IncludeStreamUsage *bool
	UserAgent          string
	Retry              RetryPolicy
}

type RetryPolicy 

type RetryPolicy struct {
	MaxRetries    int
	InitialDelay  time.Duration
	MaxDelay      time.Duration
	RetryAfterCap time.Duration
	Sleep         func(context.Context, time.Duration) error
	Jitter        func(time.Duration) time.Duration
}

type SigV4Authenticator 

type SigV4Authenticator struct {
	Region      string
	Service     string
	Credentials AWSCredentialsProvider
	Clock       func() time.Time
}

SigV4Authenticator signs the final HTTP request, including its escaped path and encoded body. Service defaults to "bedrock". Clock is intended for deterministic tests; production callers should leave it nil.

func (SigV4Authenticator) Apply 

func (a SigV4Authenticator) Apply(ctx context.Context, request *http.Request) error

type Stream 

type Stream struct {
	// contains filtered or unexported fields
}

func (*Stream) Close 

func (s *Stream) Close() error

func (*Stream) Collect 

func (s *Stream) Collect(ctx context.Context) (llmprotocol.Response, error)

func (*Stream) Metadata 

func (s *Stream) Metadata() CallMetadata

func (*Stream) Next 

func (s *Stream) Next(ctx context.Context) (llmprotocol.StreamEvent, error)

func (*Stream) Response 

func (s *Stream) Response() llmprotocol.Response

type UnauthorizedRecoverer added in v0.3.0 

type UnauthorizedRecoverer interface {
	RecoverUnauthorized(context.Context) error
}

UnauthorizedRecoverer is an optional capability implemented by dynamic authenticators. When an upstream rejects a request with HTTP 401, Client asks the authenticator to refresh or otherwise recover its credentials, then replays the request exactly once. Static authenticators need only implement Authenticator.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.