Help us shape the future of Go! Take the 2026 Go Developer Survey

inference

package
v0.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 20, 2026 License: MIT Imports: 25 Imported by: 0

Documentation

Overview

Package inference runs bounded, reusable text-generation jobs through Questlock. The broker stores coordination and results; API credentials stay in the worker.

Index

Constants

View Source 
const SGLangRuntimeVersion = "0.5.20"

Variables

View Source 
var (
	ErrInFlight       = errors.New("request is pending or has an uncertain outcome; inspect its artifact before deliberately changing revision to retry")
	ErrIncomplete     = errors.New("provider returned incomplete output; result is saved but cannot be reused; inspect before changing revision")
	ErrBudgetExceeded = errors.New("request exceeds the configured token or cost budget")
)

Functions

func BoundedTextPrompt added in v0.9.0 

func BoundedTextPrompt(req Request) string

BoundedTextPrompt serializes the explicit user request for text-only native execution and matched baselines. Harness/system context can still differ.

func SelfHosted added in v0.5.0 

func SelfHosted(s Spec) bool

SelfHosted reports the explicit no-provider-token-metering contract. It does not assert that hardware, electricity, infrastructure or operator time is free.

func ValidateSGLangConfig added in v0.5.0 

func ValidateSGLangConfig(cfg SGLangConfig) error

Types

type Broker 

type Broker interface {
	Get(context.Context, string) (protocol.Artifact, error)
	Publish(context.Context, protocol.PublishRequest) (protocol.PublishResult, error)
}

type ContextItem 

type ContextItem struct {
	ID       string `json:"id"`
	Text     string `json:"text"`
	Required bool   `json:"required,omitempty"`
	Priority int    `json:"priority,omitempty"`
	Stable   bool   `json:"stable,omitempty"`
}

type GenerationTiming added in v0.3.0 

type GenerationTiming struct {
	Streaming    bool     `json:"streaming"`
	FirstTokenMS *float64 `json:"first_content_ms,omitempty"`
	OutputMS     *float64 `json:"output_content_ms,omitempty"`
	Basis        string   `json:"basis"`
}

GenerationTiming describes streamed local content chunks observed by the client. FirstTokenMS includes metadata/network time from Generate entry. OutputMS is the first-to-last content-chunk interval, not pure compute time. Missing measurements are unknown; a one-chunk interval can validly be zero.

type HTTPProvider 

type HTTPProvider struct {
	// contains filtered or unexported fields
}

HTTPProvider sends one text-only request to OpenAI Responses or Claude Messages. It does not retry, follow redirects, use environment proxies, or fall back to another model. Credentials are held only by this worker.

func NewHTTPProvider 

func NewHTTPProvider(provider, apiKey, baseURL string) (*HTTPProvider, error)

NewHTTPProvider accepts openai, anthropic, or claude. An empty baseURL uses the provider's public /v1 endpoint. Custom endpoints require HTTPS, except loopback HTTP endpoints for local gateways and tests.

func (*HTTPProvider) CountInputTokens 

func (p *HTTPProvider) CountInputTokens(ctx context.Context, req Request) (int64, error)

CountInputTokens uses the upstream counting endpoint and sends the same text and roles as generation, without unsupported generation-only fields.

func (*HTTPProvider) Generate 

func (p *HTTPProvider) Generate(ctx context.Context, req Request) (Response, error)

func (*HTTPProvider) Identity 

func (p *HTTPProvider) Identity() string

Identity excludes credentials while keeping custom gateways isolated from each other and from the provider's public endpoint in result fingerprints.

type Measurement added in v0.3.0 

type Measurement struct {
	GenerationTiming       *GenerationTiming  `json:"generation_timing,omitempty"`
	StartedAt              time.Time          `json:"started_at"`
	EndedAt                time.Time          `json:"ended_at"`
	TotalMS                float64            `json:"total_ms"`
	PhaseMS                map[string]float64 `json:"phase_ms"`
	Outcome                string             `json:"outcome"`
	Fingerprint            string             `json:"fingerprint,omitempty"`
	ContextBytes           int                `json:"context_bytes"`
	TotalContextBytes      int                `json:"total_context_bytes"`
	SelectedContext        int                `json:"selected_context"`
	OmittedContext         int                `json:"omitted_context"`
	GenerationCalled       bool               `json:"generation_called"`
	KnownGenerationUsage   bool               `json:"known_generation_usage"`
	GenerationUsage        *Usage             `json:"generation_usage,omitempty"`
	GenerationStartedAt    time.Time          `json:"generation_started_at,omitzero"`
	GenerationEndedAt      time.Time          `json:"generation_ended_at,omitzero"`
	ProviderSetupMS        float64            `json:"provider_setup_ms,omitempty"`
	ProviderSetupStartedAt time.Time          `json:"provider_setup_started_at,omitzero"`
	ProviderSetupEndedAt   time.Time          `json:"provider_setup_ended_at,omitzero"`
}

Measurement describes this invocation, never the cached answer's original latency. It is returned to the caller and is not stored in the job, included in its fingerprint, or sent anywhere automatically. Complete and durable describe protocol state, not answer quality.

type OllamaProvider 

type OllamaProvider struct {
	// contains filtered or unexported fields
}

OllamaProvider runs installed model weights through a literal loopback HTTP endpoint. It does not sign in, pull models, send keys, or use cloud models. A provider captures one installed model snapshot; recreate it to refresh a mutable model alias after its weights or template change.

func NewOllamaProvider 

func NewOllamaProvider(baseURL, model string) (*OllamaProvider, error)

NewOllamaProvider checks that model is already installed and advertises local text completion. baseURL defaults to http://127.0.0.1:11434; literal loopback HTTP is required so a local-mode configuration cannot use a hosted endpoint or depend on DNS resolution.

func NewOllamaProviderContext 

func NewOllamaProviderContext(ctx context.Context, baseURL, model string) (*OllamaProvider, error)

NewOllamaProviderContext includes installed-model discovery in the caller's cancellation and deadline. Discovery also has a 15-second maximum of its own.

func (*OllamaProvider) CountInputTokens 

func (p *OllamaProvider) CountInputTokens(ctx context.Context, req Request) (int64, error)

CountInputTokens is a conservative local policy ESTIMATE, not an Ollama tokenizer API or a tokenizer-specific proof. It allocates one token per byte of requested text and model template/system, plus 512 tokens of overhead. Metadata reads do not run inference; reported usage replaces the estimate after generation. Context and generation caps still govern local resources.

func (*OllamaProvider) Generate 

func (p *OllamaProvider) Generate(ctx context.Context, req Request) (Response, error)

func (*OllamaProvider) Identity 

func (p *OllamaProvider) Identity() string

func (*OllamaProvider) ModelDigest added in v0.3.0 

func (p *OllamaProvider) ModelDigest() string

ModelDigest exposes the installed weights identity without endpoint, prompt, template or system content. It performs no inference or metadata request.

type OpenRouterConfig added in v0.5.0 

type OpenRouterConfig struct {
	UpstreamProvider     string `json:"upstream_provider"`
	InputTokenUpperBound int64  `json:"input_token_upper_bound"`
	CacheWriteUsage      string `json:"cache_write_usage"`
}

OpenRouterConfig describes one explicitly approved route. The input bound is supplied by the operator: OpenRouter has no documented preflight counting API for arbitrary models. It is a reservation bound, not a measured token count. CacheWriteUsage must be "required" for a route with cache-write pricing, or "not_applicable" for a route whose cache writes have no separate token charge.

type OpenRouterProvider added in v0.5.0 

type OpenRouterProvider struct {
	// contains filtered or unexported fields
}

OpenRouterProvider implements a deliberately narrow Chat Completions route: one model, one upstream, no fallbacks, retries, tools, or streaming. It does not translate OpenAI Responses cache breakpoints into gateway parameters.

func NewOpenRouterProvider added in v0.5.0 

func NewOpenRouterProvider(apiKey, baseURL string, config OpenRouterConfig) (*OpenRouterProvider, error)

func (*OpenRouterProvider) CountInputTokens added in v0.5.0 

func (p *OpenRouterProvider) CountInputTokens(ctx context.Context, req Request) (int64, error)

CountInputTokens returns the explicit operator-approved reservation bound. No tokenizer equivalence, exact preflight count, or upstream call is implied.

func (*OpenRouterProvider) Generate added in v0.5.0 

func (p *OpenRouterProvider) Generate(ctx context.Context, req Request) (Response, error)

func (*OpenRouterProvider) Identity added in v0.5.0 

func (p *OpenRouterProvider) Identity() string

type Plan 

type Plan struct {
	Fingerprint       string   `json:"fingerprint"`
	Request           Request  `json:"request"`
	Selected          []string `json:"selected_context"`
	Omitted           []string `json:"omitted_context"`
	ContextBytes      int      `json:"context_bytes"`
	TotalContextBytes int      `json:"total_context_bytes"`
}

func BuildPlan 

func BuildPlan(s Spec) (Plan, error)

BuildPlan selects whole context chunks without a model call. Required chunks must all fit. Optional chunks use explicit descending priority, then ID. The resulting prefix and input are a reviewable snapshot, never a hidden summary.

type Prices 

type Prices struct {
	Input      float64 `json:"input_per_million"`
	CacheRead  float64 `json:"cache_read_per_million"`
	CacheWrite float64 `json:"cache_write_per_million"`
	Output     float64 `json:"output_per_million"`
}

Prices are explicit USD per million tokens, not a built-in or live price list. CacheWrite must cover the cache policy in use, including any write premium.

type Provider 

type Provider interface {
	// Identity distinguishes upstream endpoints in durable request fingerprints.
	Identity() string
	CountInputTokens(context.Context, Request) (int64, error)
	Generate(context.Context, Request) (Response, error)
}

type Request 

type Request struct {
	OpenRouter      *OpenRouterConfig `json:"openrouter,omitempty"`
	SGLang          *SGLangConfig     `json:"sglang,omitempty"`
	Stream          bool              `json:"stream,omitempty"`
	Provider        string            `json:"provider"`
	Model           string            `json:"model"`
	ReasoningEffort string            `json:"reasoning_effort,omitempty"`
	ContextWindow   int64             `json:"context_window,omitempty"` // local runtime only
	Prefix          string            `json:"prefix"`
	Input           string            `json:"input"`
	MaxOutputTokens int64             `json:"max_output_tokens"`
	CacheMode       string            `json:"cache_mode"`
}

Request is provider-independent, single-turn text generation without tools. Prefix contains approved instructions and stable reference material; Input contains changing reference material and the user's task.

type Response 

type Response struct {
	RuntimeCache *RuntimeCacheObservation `json:"runtime_cache,omitempty"`
	Timing       *GenerationTiming        `json:"timing,omitempty"`
	ID           string                   `json:"id"`
	Model        string                   `json:"model"`
	Text         string                   `json:"text"`
	Complete     bool                     `json:"complete"`
	StopReason   string                   `json:"stop_reason"`
	Usage        Usage                    `json:"usage"`
}

type Result 

type Result struct {
	GenerationRuntimeCache *RuntimeCacheObservation `json:"generation_runtime_cache,omitempty"`
	GenerationTiming       *GenerationTiming        `json:"generation_timing,omitempty"`
	Measurement            *Measurement             `json:"measurement,omitempty"`
	Fingerprint            string                   `json:"fingerprint"`
	Response               Response                 `json:"response"`
	Reused                 bool                     `json:"reused"`
	Durable                bool                     `json:"durable"`
	GenerationUsage        Usage                    `json:"generation_usage"`
	ThisRunUsage           Usage                    `json:"this_run_usage"`
	AvoidedUsage           Usage                    `json:"avoided_usage"`
	CostUSD                *float64                 `json:"cost_usd,omitempty"`
	AvoidedCostUSD         *float64                 `json:"avoided_cost_usd,omitempty"`
	CreatedAt              time.Time                `json:"created_at"`
	ExpiresAt              time.Time                `json:"expires_at"`
}

type Runner 

type Runner struct {
	Broker   Broker
	Provider Provider
	Now      func() time.Time
}

Runner coordinates workers via the existing broker API. Now may be set by tests; production uses the UTC wall clock. A pending claim never expires into another paid call: a crashed worker may already have incurred a charge.

func (Runner) Run 

func (r Runner) Run(ctx context.Context, s Spec) (result Result, runErr error)

type RuntimeCacheObservation added in v0.5.0 

type RuntimeCacheObservation struct {
	CachedTokens   *int64 `json:"cached_tokens,omitempty"`
	DeviceTokens   *int64 `json:"device_tokens,omitempty"`
	HostTokens     *int64 `json:"host_tokens,omitempty"`
	StorageTokens  *int64 `json:"storage_tokens,omitempty"`
	StorageBackend string `json:"storage_backend,omitempty"`
	Profile        string `json:"profile"`
	RuntimeVersion string `json:"runtime_version"`
	Evidence       string `json:"evidence"`
}

RuntimeCacheObservation describes compute reuse reported by a serving runtime. These counters are not billing cache buckets or proof of an invoice discount. A nil counter means unreported, rather than zero. Historical observations on a reused result describe its original generation, not a new runtime request.

type SGLangCapabilities added in v0.5.0 

type SGLangCapabilities struct {
	Profile               string `json:"profile"`
	RuntimeVersion        string `json:"runtime_version"`
	GenerationSupported   bool   `json:"generation_supported"`
	RuntimeIdentitySHA256 string `json:"runtime_identity_sha256"`
	Observed              bool   `json:"observed"`
	Reason                string `json:"reason,omitempty"`
}

SGLangCapabilities contains allowlisted readback facts, without endpoints, prompts, model paths, launch commands, or credentials. Observed configuration is not proof that a particular offload kernel ran or improved performance.

type SGLangConfig added in v0.5.0 

type SGLangConfig struct {
	DeploymentID   string `json:"deployment_id"`
	CacheNamespace string `json:"cache_namespace"`
	Profile        string `json:"profile"`
	SelfHosted     bool   `json:"self_hosted"`
	MaxInputTokens int64  `json:"max_input_tokens"`
}

SGLangConfig identifies an explicitly operated, immutable serving deployment. DeploymentID is the operator's model/configuration revision, not a weight digest proved by the HTTP API. Change it whenever that deployment changes.

type SGLangProvider added in v0.5.0 

type SGLangProvider struct {
	// contains filtered or unexported fields
}

SGLangProvider implements the pinned native single-turn, text-only contract. SGLang owns its KV tensors and storage. Questlock performs no administration, model download, runtime reconfiguration, retry, or PD bootstrap orchestration.

func NewSGLangProviderContext added in v0.5.0 

func NewSGLangProviderContext(ctx context.Context, apiKey, baseURL, model string, cfg SGLangConfig) (*SGLangProvider, error)

NewSGLangProviderContext observes an already-running SGLang v0.5.20 server. Literal loopback HTTP permits no key; remote HTTPS requires an explicit key. Constructor readbacks run no generation and include no admin writes.

func (*SGLangProvider) Capabilities added in v0.5.0 

func (p *SGLangProvider) Capabilities() SGLangCapabilities

func (*SGLangProvider) CountInputTokens added in v0.5.0 

func (p *SGLangProvider) CountInputTokens(ctx context.Context, req Request) (int64, error)

func (*SGLangProvider) Generate added in v0.5.0 

func (p *SGLangProvider) Generate(ctx context.Context, req Request) (Response, error)

func (*SGLangProvider) Identity added in v0.5.0 

func (p *SGLangProvider) Identity() string

func (*SGLangProvider) RefreshIdentity added in v0.5.0 

func (p *SGLangProvider) RefreshIdentity(ctx context.Context) error

RefreshIdentity is used before a Runner result lookup as well as around tokenization/generation. An existing provider never silently adopts a new model or restart; construct a fresh provider for that new deployment.

type Spec 

type Spec struct {
	OpenRouter      *OpenRouterConfig `json:"openrouter,omitempty"`
	SGLang          *SGLangConfig     `json:"sglang,omitempty"`
	Stream          bool              `json:"stream,omitempty"` // Optional local Ollama content streaming.
	Task            string            `json:"task"`
	Revision        string            `json:"revision"`
	Provider        string            `json:"provider"`
	Model           string            `json:"model"`
	ReasoningEffort string            `json:"reasoning_effort,omitempty"`
	Instructions    string            `json:"instructions"`
	Input           string            `json:"input"`
	Context         []ContextItem     `json:"context,omitempty"`
	MaxContextBytes int               `json:"max_context_bytes"`
	MaxInputTokens  int64             `json:"max_input_tokens"`
	MaxOutputTokens int64             `json:"max_output_tokens"`
	CacheMode       string            `json:"cache_mode,omitempty"` // default or prefix
	ReuseForSeconds int64             `json:"reuse_for_seconds,omitempty"`
	Prices          *Prices           `json:"prices,omitempty"`
	MaxCostUSD      float64           `json:"max_cost_usd,omitempty"`
	DailyBudgetUSD  float64           `json:"daily_budget_usd,omitempty"`
}

Spec is an explicit snapshot of a task. Reuse is opt-in and should only be enabled for tasks whose answer remains valid for the supplied revision.

type Usage 

type Usage struct {
	InputTokens         int64 `json:"input_tokens"`
	UncachedInputTokens int64 `json:"uncached_input_tokens"`
	CacheReadTokens     int64 `json:"cache_read_tokens"`
	CacheWriteTokens    int64 `json:"cache_write_tokens"`
	OutputTokens        int64 `json:"output_tokens"`
	ReasoningTokens     int64 `json:"reasoning_tokens,omitempty"`
}

Usage partitions input tokens into uncached, cache reads, and cache writes. InputTokens is their sum on all providers. Reasoning is already included in OutputTokens, and must not be charged a second time.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.