Help us shape the future of Go! Take the 2026 Go Developer Survey

inference

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 13, 2026 License: MIT Imports: 21 Imported by: 0

Documentation

Overview

Package inference runs bounded, reusable text-generation jobs through Questlock. The broker stores coordination and results; API credentials stay in the worker.

Index

Constants

This section is empty.

Variables

View Source 
var (
	ErrInFlight       = errors.New("request is pending or has an uncertain outcome; inspect its artifact before deliberately changing revision to retry")
	ErrIncomplete     = errors.New("provider returned incomplete output; result is saved but cannot be reused; inspect before changing revision")
	ErrBudgetExceeded = errors.New("request exceeds the configured token or cost budget")
)

Functions

This section is empty.

Types

type Broker 

type Broker interface {
	Get(context.Context, string) (protocol.Artifact, error)
	Publish(context.Context, protocol.PublishRequest) (protocol.PublishResult, error)
}

type ContextItem 

type ContextItem struct {
	ID       string `json:"id"`
	Text     string `json:"text"`
	Required bool   `json:"required,omitempty"`
	Priority int    `json:"priority,omitempty"`
	Stable   bool   `json:"stable,omitempty"`
}

type HTTPProvider 

type HTTPProvider struct {
	// contains filtered or unexported fields
}

HTTPProvider sends one text-only request to OpenAI Responses or Claude Messages. It does not retry, follow redirects, use environment proxies, or fall back to another model. Credentials are held only by this worker.

func NewHTTPProvider 

func NewHTTPProvider(provider, apiKey, baseURL string) (*HTTPProvider, error)

NewHTTPProvider accepts openai, anthropic, or claude. An empty baseURL uses the provider's public /v1 endpoint. Custom endpoints require HTTPS, except loopback HTTP endpoints for local gateways and tests.

func (*HTTPProvider) CountInputTokens 

func (p *HTTPProvider) CountInputTokens(ctx context.Context, req Request) (int64, error)

CountInputTokens uses the upstream counting endpoint and sends the same text and roles as generation, without unsupported generation-only fields.

func (*HTTPProvider) Generate 

func (p *HTTPProvider) Generate(ctx context.Context, req Request) (Response, error)

func (*HTTPProvider) Identity 

func (p *HTTPProvider) Identity() string

Identity excludes credentials while keeping custom gateways isolated from each other and from the provider's public endpoint in result fingerprints.

type OllamaProvider 

type OllamaProvider struct {
	// contains filtered or unexported fields
}

OllamaProvider runs installed model weights through a literal loopback HTTP endpoint. It does not sign in, pull models, send keys, or use cloud models. A provider captures one installed model snapshot; recreate it to refresh a mutable model alias after its weights or template change.

func NewOllamaProvider 

func NewOllamaProvider(baseURL, model string) (*OllamaProvider, error)

NewOllamaProvider checks that model is already installed and advertises local text completion. baseURL defaults to http://127.0.0.1:11434; literal loopback HTTP is required so a local-mode configuration cannot use a hosted endpoint or depend on DNS resolution.

func NewOllamaProviderContext 

func NewOllamaProviderContext(ctx context.Context, baseURL, model string) (*OllamaProvider, error)

NewOllamaProviderContext includes installed-model discovery in the caller's cancellation and deadline. Discovery also has a 15-second maximum of its own.

func (*OllamaProvider) CountInputTokens 

func (p *OllamaProvider) CountInputTokens(ctx context.Context, req Request) (int64, error)

CountInputTokens is a conservative local policy ESTIMATE, not an Ollama tokenizer API or a tokenizer-specific proof. It allocates one token per byte of requested text and model template/system, plus 512 tokens of overhead. Metadata reads do not run inference; reported usage replaces the estimate after generation. Context and generation caps still govern local resources.

func (*OllamaProvider) Generate 

func (p *OllamaProvider) Generate(ctx context.Context, req Request) (Response, error)

func (*OllamaProvider) Identity 

func (p *OllamaProvider) Identity() string

type Plan 

type Plan struct {
	Fingerprint       string   `json:"fingerprint"`
	Request           Request  `json:"request"`
	Selected          []string `json:"selected_context"`
	Omitted           []string `json:"omitted_context"`
	ContextBytes      int      `json:"context_bytes"`
	TotalContextBytes int      `json:"total_context_bytes"`
}

func BuildPlan 

func BuildPlan(s Spec) (Plan, error)

BuildPlan selects whole context chunks without a model call. Required chunks must all fit. Optional chunks use explicit descending priority, then ID. The resulting prefix and input are a reviewable snapshot, never a hidden summary.

type Prices 

type Prices struct {
	Input      float64 `json:"input_per_million"`
	CacheRead  float64 `json:"cache_read_per_million"`
	CacheWrite float64 `json:"cache_write_per_million"`
	Output     float64 `json:"output_per_million"`
}

Prices are explicit USD per million tokens, not a built-in or live price list. CacheWrite must cover the cache policy in use, including any write premium.

type Provider 

type Provider interface {
	// Identity distinguishes upstream endpoints in durable request fingerprints.
	Identity() string
	CountInputTokens(context.Context, Request) (int64, error)
	Generate(context.Context, Request) (Response, error)
}

type Request 

type Request struct {
	Provider        string `json:"provider"`
	Model           string `json:"model"`
	ReasoningEffort string `json:"reasoning_effort,omitempty"`
	ContextWindow   int64  `json:"context_window,omitempty"` // local runtime only
	Prefix          string `json:"prefix"`
	Input           string `json:"input"`
	MaxOutputTokens int64  `json:"max_output_tokens"`
	CacheMode       string `json:"cache_mode"`
}

Request is provider-independent, single-turn text generation without tools. Prefix contains approved instructions and stable reference material; Input contains changing reference material and the user's task.

type Response 

type Response struct {
	ID         string `json:"id"`
	Model      string `json:"model"`
	Text       string `json:"text"`
	Complete   bool   `json:"complete"`
	StopReason string `json:"stop_reason"`
	Usage      Usage  `json:"usage"`
}

type Result 

type Result struct {
	Fingerprint     string    `json:"fingerprint"`
	Response        Response  `json:"response"`
	Reused          bool      `json:"reused"`
	Durable         bool      `json:"durable"`
	GenerationUsage Usage     `json:"generation_usage"`
	ThisRunUsage    Usage     `json:"this_run_usage"`
	AvoidedUsage    Usage     `json:"avoided_usage"`
	CostUSD         *float64  `json:"cost_usd,omitempty"`
	AvoidedCostUSD  *float64  `json:"avoided_cost_usd,omitempty"`
	CreatedAt       time.Time `json:"created_at"`
	ExpiresAt       time.Time `json:"expires_at"`
}

type Runner 

type Runner struct {
	Broker   Broker
	Provider Provider
	Now      func() time.Time
}

Runner coordinates workers via the existing broker API. Now may be set by tests; production uses the UTC wall clock. A pending claim never expires into another paid call: a crashed worker may already have incurred a charge.

func (Runner) Run 

func (r Runner) Run(ctx context.Context, s Spec) (Result, error)

type Spec 

type Spec struct {
	Task            string        `json:"task"`
	Revision        string        `json:"revision"`
	Provider        string        `json:"provider"`
	Model           string        `json:"model"`
	ReasoningEffort string        `json:"reasoning_effort,omitempty"`
	Instructions    string        `json:"instructions"`
	Input           string        `json:"input"`
	Context         []ContextItem `json:"context,omitempty"`
	MaxContextBytes int           `json:"max_context_bytes"`
	MaxInputTokens  int64         `json:"max_input_tokens"`
	MaxOutputTokens int64         `json:"max_output_tokens"`
	CacheMode       string        `json:"cache_mode,omitempty"` // default or prefix
	ReuseForSeconds int64         `json:"reuse_for_seconds,omitempty"`
	Prices          *Prices       `json:"prices,omitempty"`
	MaxCostUSD      float64       `json:"max_cost_usd,omitempty"`
	DailyBudgetUSD  float64       `json:"daily_budget_usd,omitempty"`
}

Spec is an explicit snapshot of a task. Reuse is opt-in and should only be enabled for tasks whose answer remains valid for the supplied revision.

type Usage 

type Usage struct {
	InputTokens         int64 `json:"input_tokens"`
	UncachedInputTokens int64 `json:"uncached_input_tokens"`
	CacheReadTokens     int64 `json:"cache_read_tokens"`
	CacheWriteTokens    int64 `json:"cache_write_tokens"`
	OutputTokens        int64 `json:"output_tokens"`
	ReasoningTokens     int64 `json:"reasoning_tokens,omitempty"`
}

Usage partitions input tokens into uncached, cache reads, and cache writes. InputTokens is their sum on all providers. Reasoning is already included in OutputTokens, and must not be charged a second time.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.