Documentation
¶
Overview ¶
Package inference runs bounded, reusable text-generation jobs through Questlock. The broker stores coordination and results; API credentials stay in the worker.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( ErrInFlight = errors.New("request is pending or has an uncertain outcome; inspect its artifact before deliberately changing revision to retry") ErrIncomplete = errors.New("provider returned incomplete output; result is saved but cannot be reused; inspect before changing revision") ErrBudgetExceeded = errors.New("request exceeds the configured token or cost budget") )
Functions ¶
This section is empty.
Types ¶
type ContextItem ¶
type GenerationTiming ¶ added in v0.3.0
type GenerationTiming struct {
Streaming bool `json:"streaming"`
FirstTokenMS *float64 `json:"first_content_ms,omitempty"`
OutputMS *float64 `json:"output_content_ms,omitempty"`
Basis string `json:"basis"`
}
GenerationTiming describes streamed local content chunks observed by the client. FirstTokenMS includes metadata/network time from Generate entry. OutputMS is the first-to-last content-chunk interval, not pure compute time. Missing measurements are unknown; a one-chunk interval can validly be zero.
type HTTPProvider ¶
type HTTPProvider struct {
// contains filtered or unexported fields
}
HTTPProvider sends one text-only request to OpenAI Responses or Claude Messages. It does not retry, follow redirects, use environment proxies, or fall back to another model. Credentials are held only by this worker.
func NewHTTPProvider ¶
func NewHTTPProvider(provider, apiKey, baseURL string) (*HTTPProvider, error)
NewHTTPProvider accepts openai, anthropic, or claude. An empty baseURL uses the provider's public /v1 endpoint. Custom endpoints require HTTPS, except loopback HTTP endpoints for local gateways and tests.
func (*HTTPProvider) CountInputTokens ¶
CountInputTokens uses the upstream counting endpoint and sends the same text and roles as generation, without unsupported generation-only fields.
func (*HTTPProvider) Identity ¶
func (p *HTTPProvider) Identity() string
Identity excludes credentials while keeping custom gateways isolated from each other and from the provider's public endpoint in result fingerprints.
type Measurement ¶ added in v0.3.0
type Measurement struct {
GenerationTiming *GenerationTiming `json:"generation_timing,omitempty"`
StartedAt time.Time `json:"started_at"`
EndedAt time.Time `json:"ended_at"`
TotalMS float64 `json:"total_ms"`
PhaseMS map[string]float64 `json:"phase_ms"`
Outcome string `json:"outcome"`
Fingerprint string `json:"fingerprint,omitempty"`
ContextBytes int `json:"context_bytes"`
TotalContextBytes int `json:"total_context_bytes"`
SelectedContext int `json:"selected_context"`
OmittedContext int `json:"omitted_context"`
GenerationCalled bool `json:"generation_called"`
KnownGenerationUsage bool `json:"known_generation_usage"`
GenerationUsage *Usage `json:"generation_usage,omitempty"`
GenerationStartedAt time.Time `json:"generation_started_at,omitzero"`
GenerationEndedAt time.Time `json:"generation_ended_at,omitzero"`
ProviderSetupMS float64 `json:"provider_setup_ms,omitempty"`
ProviderSetupStartedAt time.Time `json:"provider_setup_started_at,omitzero"`
ProviderSetupEndedAt time.Time `json:"provider_setup_ended_at,omitzero"`
}
Measurement describes this invocation, never the cached answer's original latency. It is returned to the caller and is not stored in the job, included in its fingerprint, or sent anywhere automatically. Complete and durable describe protocol state, not answer quality.
type OllamaProvider ¶
type OllamaProvider struct {
// contains filtered or unexported fields
}
OllamaProvider runs installed model weights through a literal loopback HTTP endpoint. It does not sign in, pull models, send keys, or use cloud models. A provider captures one installed model snapshot; recreate it to refresh a mutable model alias after its weights or template change.
func NewOllamaProvider ¶
func NewOllamaProvider(baseURL, model string) (*OllamaProvider, error)
NewOllamaProvider checks that model is already installed and advertises local text completion. baseURL defaults to http://127.0.0.1:11434; literal loopback HTTP is required so a local-mode configuration cannot use a hosted endpoint or depend on DNS resolution.
func NewOllamaProviderContext ¶
func NewOllamaProviderContext(ctx context.Context, baseURL, model string) (*OllamaProvider, error)
NewOllamaProviderContext includes installed-model discovery in the caller's cancellation and deadline. Discovery also has a 15-second maximum of its own.
func (*OllamaProvider) CountInputTokens ¶
CountInputTokens is a conservative local policy ESTIMATE, not an Ollama tokenizer API or a tokenizer-specific proof. It allocates one token per byte of requested text and model template/system, plus 512 tokens of overhead. Metadata reads do not run inference; reported usage replaces the estimate after generation. Context and generation caps still govern local resources.
func (*OllamaProvider) Identity ¶
func (p *OllamaProvider) Identity() string
func (*OllamaProvider) ModelDigest ¶ added in v0.3.0
func (p *OllamaProvider) ModelDigest() string
ModelDigest exposes the installed weights identity without endpoint, prompt, template or system content. It performs no inference or metadata request.
type Plan ¶
type Prices ¶
type Prices struct {
Input float64 `json:"input_per_million"`
CacheRead float64 `json:"cache_read_per_million"`
CacheWrite float64 `json:"cache_write_per_million"`
Output float64 `json:"output_per_million"`
}
Prices are explicit USD per million tokens, not a built-in or live price list. CacheWrite must cover the cache policy in use, including any write premium.
type Request ¶
type Request struct {
Stream bool `json:"stream,omitempty"`
Provider string `json:"provider"`
Model string `json:"model"`
ReasoningEffort string `json:"reasoning_effort,omitempty"`
ContextWindow int64 `json:"context_window,omitempty"` // local runtime only
Prefix string `json:"prefix"`
Input string `json:"input"`
MaxOutputTokens int64 `json:"max_output_tokens"`
CacheMode string `json:"cache_mode"`
}
Request is provider-independent, single-turn text generation without tools. Prefix contains approved instructions and stable reference material; Input contains changing reference material and the user's task.
type Result ¶
type Result struct {
GenerationTiming *GenerationTiming `json:"generation_timing,omitempty"`
Measurement *Measurement `json:"measurement,omitempty"`
Fingerprint string `json:"fingerprint"`
Response Response `json:"response"`
Reused bool `json:"reused"`
Durable bool `json:"durable"`
GenerationUsage Usage `json:"generation_usage"`
ThisRunUsage Usage `json:"this_run_usage"`
AvoidedUsage Usage `json:"avoided_usage"`
CostUSD *float64 `json:"cost_usd,omitempty"`
AvoidedCostUSD *float64 `json:"avoided_cost_usd,omitempty"`
CreatedAt time.Time `json:"created_at"`
ExpiresAt time.Time `json:"expires_at"`
}
type Runner ¶
Runner coordinates workers via the existing broker API. Now may be set by tests; production uses the UTC wall clock. A pending claim never expires into another paid call: a crashed worker may already have incurred a charge.
type Spec ¶
type Spec struct {
Stream bool `json:"stream,omitempty"` // Optional local Ollama content streaming.
Task string `json:"task"`
Revision string `json:"revision"`
Provider string `json:"provider"`
Model string `json:"model"`
ReasoningEffort string `json:"reasoning_effort,omitempty"`
Instructions string `json:"instructions"`
Input string `json:"input"`
Context []ContextItem `json:"context,omitempty"`
MaxContextBytes int `json:"max_context_bytes"`
MaxInputTokens int64 `json:"max_input_tokens"`
MaxOutputTokens int64 `json:"max_output_tokens"`
CacheMode string `json:"cache_mode,omitempty"` // default or prefix
ReuseForSeconds int64 `json:"reuse_for_seconds,omitempty"`
Prices *Prices `json:"prices,omitempty"`
MaxCostUSD float64 `json:"max_cost_usd,omitempty"`
DailyBudgetUSD float64 `json:"daily_budget_usd,omitempty"`
}
Spec is an explicit snapshot of a task. Reuse is opt-in and should only be enabled for tasks whose answer remains valid for the supplied revision.
type Usage ¶
type Usage struct {
InputTokens int64 `json:"input_tokens"`
UncachedInputTokens int64 `json:"uncached_input_tokens"`
CacheReadTokens int64 `json:"cache_read_tokens"`
CacheWriteTokens int64 `json:"cache_write_tokens"`
OutputTokens int64 `json:"output_tokens"`
ReasoningTokens int64 `json:"reasoning_tokens,omitempty"`
}
Usage partitions input tokens into uncached, cache reads, and cache writes. InputTokens is their sum on all providers. Reasoning is already included in OutputTokens, and must not be charged a second time.
Source Files