Documentation
¶
Overview ¶
Package llmgateway is the single chokepoint through which every LLM call in the process must pass. Its job is to bound provider concurrency (--max-concurrent-llm-requests) across every code path (direct analysis, queue-driven analysis, digest dedupe, digest summary) so the flag actually means what it says.
A Gateway owns one semaphore (`chan struct{}`) sized at construction time. Generate and Stream acquire a slot, invoke the provider, and release in a deferred call so slot leaks can't happen on panic or context cancel. One call == one slot. A 5-task analysis pipeline holds the slot 5 separate times, leaving gaps between tasks available to other callers.
Index ¶
- func RunIDFromContext(ctx context.Context) string
- func WithRunID(ctx context.Context, runID string) context.Context
- type CallOption
- type CallRecord
- type Gateway
- func (g *Gateway) Generate(ctx context.Context, p llmprovider.Provider, prompt string, opts ...CallOption) (string, error)
- func (g *Gateway) MaxConcurrent() int
- func (g *Gateway) SetRecorder(r Recorder)
- func (g *Gateway) Stats() Stats
- func (g *Gateway) Stream(ctx context.Context, p llmprovider.ChatModelProvider, ...) (resp string, err error)
- type Recorder
- type Stats
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func RunIDFromContext ¶ added in v0.4.0
RunIDFromContext returns the run id tagged on ctx, or "" if none.
Types ¶
type CallOption ¶
type CallOption func(*callConfig)
CallOption configures a single call (Generate or Stream).
func WithLabel ¶
func WithLabel(label string) CallOption
WithLabel attaches a human-readable label to the call for log correlation (e.g. "analyze:task=key_points", "digest:dedupe", "digest:summary").
func WithModelInfo ¶ added in v0.4.0
func WithModelInfo(providerType, modelName string) CallOption
WithModelInfo records the resolved provider type and model name for the call so the monitoring recorder can attribute tokens to a model. The gateway only sees an opaque Provider, so callers that have a resolved selection pass it here.
type CallRecord ¶ added in v0.4.0
type CallRecord struct {
RunID string
Label string
ProviderType string
ModelName string
Prompt string
Response string
Usage llmprovider.Usage
UsageKnown bool
Duration time.Duration
Err error
}
CallRecord is one completed LLM call handed to a Recorder. The gateway fills the fields it knows; provider/model come from WithModelInfo, the run id from the call context (see WithRunID).
type Gateway ¶
type Gateway struct {
// contains filtered or unexported fields
}
Gateway throttles LLM calls globally.
func New ¶
New creates a Gateway that allows at most maxConcurrent simultaneous LLM calls. maxConcurrent < 1 is coerced to 1.
func (*Gateway) Generate ¶
func (g *Gateway) Generate( ctx context.Context, p llmprovider.Provider, prompt string, opts ...CallOption, ) (string, error)
Generate makes a one-shot (non-streaming) LLM call through the gateway.
func (*Gateway) MaxConcurrent ¶
MaxConcurrent returns the configured cap.
func (*Gateway) SetRecorder ¶ added in v0.4.0
SetRecorder installs r as the gateway's call recorder. Passing nil disables recording. Not safe to call concurrently with in-flight calls; set once at construction.
func (*Gateway) Stream ¶
func (g *Gateway) Stream( ctx context.Context, p llmprovider.ChatModelProvider, messages []*schema.Message, onChunk func(chunk *schema.Message) error, opts ...CallOption, ) (resp string, err error)
Stream opens a streaming LLM call through the gateway, drains the reader, and invokes onChunk for each message chunk as it arrives. The slot is held for the entire duration of the stream (reader open → io.EOF / error / cancel).
onChunk may be nil; in that case chunks are still accumulated and returned as the full response string, but no per-token callback is invoked. Returning a non-nil error from onChunk aborts the stream and is returned.
type Recorder ¶ added in v0.4.0
type Recorder interface {
Record(CallRecord)
}
Recorder receives every LLM call that passes through the gateway. It lets the monitoring layer persist calls without the gateway depending on the store or models packages (avoiding an import cycle). Record must not block.
Source Files