Help us shape the future of Go! Take the 2026 Go Developer Survey

inference

package
v0.16.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 27, 2026 License: MIT Imports: 26 Imported by: 0

Documentation

Overview

Package inference runs bounded, reusable text-generation jobs through Questlock. The broker stores coordination and results; API credentials stay in the worker.

Index

Constants

View Source 
const SGLangRuntimeVersion = "0.5.20"

Variables

View Source 
var (
	ErrInFlight       = errors.New("request is pending or has an uncertain outcome; inspect its artifact before deliberately changing revision to retry")
	ErrIncomplete     = errors.New("provider returned incomplete output; result is saved but cannot be reused; inspect before changing revision")
	ErrBudgetExceeded = errors.New("request exceeds the configured token or cost budget")
)
View Source 
var ErrReuseUncertain = errors.New("reuse verification outcome is uncertain; inspect its receipt before retrying")

ErrReuseUncertain identifies resolver work whose outcome or accounting is not confirmed. The target stays uncertain even when its own Generate never ran: changing a retrieved source must not admit another verification automatically.

Functions

func BoundedTextPrompt added in v0.9.0 

func BoundedTextPrompt(req Request) string

BoundedTextPrompt serializes the explicit user request for text-only native execution and matched baselines. Harness/system context can still differ.

func SelfHosted added in v0.5.0 

func SelfHosted(s Spec) bool

SelfHosted reports the explicit no-provider-token-metering contract. It does not assert that hardware, electricity, infrastructure or operator time is free.

func ValidateSGLangConfig added in v0.5.0 

func ValidateSGLangConfig(cfg SGLangConfig) error

Types

type Broker 

type Broker interface {
	Get(context.Context, string) (protocol.Artifact, error)
	Publish(context.Context, protocol.PublishRequest) (protocol.PublishResult, error)
}

type BudgetDiagnostic added in v0.13.0 

type BudgetDiagnostic struct {
	LimitKind            string `json:"limit_kind"`
	ConfiguredLimit      int64  `json:"configured_limit"`
	Unit                 string `json:"unit"`
	Reason               string `json:"reason"`
	CountBasis           string `json:"count_basis"`
	GenerationCalled     bool   `json:"generation_called"`
	EstimatedInputTokens int64  `json:"estimated_input_tokens,omitempty"`
	CountedInputTokens   int64  `json:"counted_input_tokens,omitempty"`
	ReportedInputTokens  int64  `json:"reported_input_tokens,omitempty"`
	ReportedOutputTokens int64  `json:"reported_output_tokens,omitempty"`
	MaxOutputTokens      int64  `json:"max_output_tokens,omitempty"`
	ContextWindow        int64  `json:"context_window,omitempty"`
	Required             int64  `json:"required,omitempty"`
	CurrentUsage         int64  `json:"current_usage,omitempty"`
	RequestedReservation int64  `json:"requested_reservation,omitempty"`
}

BudgetDiagnostic contains only bounded numeric evidence and fixed vocabulary. It never includes provider messages, request text, model names, or endpoints. Cost limits use integer millionths of USD. CurrentUsage includes unsettled reservations in shared ledgers. Preflight counts retain their counting basis; they must not be presented as runtime-reported token usage. On a saved limit breach, GenerationCalled describes its original breach; Result.Measurement separately describes generation during the current invocation.

func BudgetDiagnosticFromError added in v0.13.0 

func BudgetDiagnosticFromError(err error) *BudgetDiagnostic

BudgetDiagnosticFromError returns a copy only when all fields belong to the safe diagnostic vocabulary. In particular, an arbitrary provider cannot smuggle its error text through a diagnostic string field.

type BudgetError added in v0.13.0 

type BudgetError struct {
	Diagnostic BudgetDiagnostic
}

BudgetError preserves errors.Is(err, ErrBudgetExceeded) and makes safe machine-readable evidence available through errors.As. Its message is fixed; callers should inspect Diagnostic rather than parse error text.

func (*BudgetError) Error added in v0.13.0 

func (e *BudgetError) Error() string

func (*BudgetError) Unwrap added in v0.13.0 

func (e *BudgetError) Unwrap() error

type ContextItem 

type ContextItem struct {
	ID       string `json:"id"`
	Text     string `json:"text"`
	Required bool   `json:"required,omitempty"`
	Priority int    `json:"priority,omitempty"`
	Stable   bool   `json:"stable,omitempty"`
}

type GenerationTiming added in v0.3.0 

type GenerationTiming struct {
	Streaming    bool     `json:"streaming"`
	FirstTokenMS *float64 `json:"first_content_ms,omitempty"`
	OutputMS     *float64 `json:"output_content_ms,omitempty"`
	Basis        string   `json:"basis"`
}

GenerationTiming describes streamed local content chunks observed by the client. FirstTokenMS includes metadata/network time from Generate entry. OutputMS is the first-to-last content-chunk interval, not pure compute time. Missing measurements are unknown; a one-chunk interval can validly be zero.

type HTTPProvider 

type HTTPProvider struct {
	// contains filtered or unexported fields
}

HTTPProvider sends one text-only request to OpenAI Responses or Claude Messages. It does not retry, follow redirects, use environment proxies, or fall back to another model. Credentials are held only by this worker.

func NewHTTPProvider 

func NewHTTPProvider(provider, apiKey, baseURL string) (*HTTPProvider, error)

NewHTTPProvider accepts openai, anthropic, or claude. An empty baseURL uses the provider's public /v1 endpoint. Custom endpoints require HTTPS, except loopback HTTP endpoints for local gateways and tests.

func (*HTTPProvider) CountInputTokens 

func (p *HTTPProvider) CountInputTokens(ctx context.Context, req Request) (int64, error)

CountInputTokens uses the upstream counting endpoint and sends the same text and roles as generation, without unsupported generation-only fields.

func (*HTTPProvider) Generate 

func (p *HTTPProvider) Generate(ctx context.Context, req Request) (Response, error)

func (*HTTPProvider) Identity 

func (p *HTTPProvider) Identity() string

Identity excludes credentials while keeping custom gateways isolated from each other and from the provider's public endpoint in result fingerprints.

func (*HTTPProvider) InputTokenCountBasis added in v0.13.0 

func (p *HTTPProvider) InputTokenCountBasis() string

type Measurement added in v0.3.0 

type Measurement struct {
	GenerationTiming       *GenerationTiming  `json:"generation_timing,omitempty"`
	StartedAt              time.Time          `json:"started_at"`
	EndedAt                time.Time          `json:"ended_at"`
	TotalMS                float64            `json:"total_ms"`
	PhaseMS                map[string]float64 `json:"phase_ms"`
	Outcome                string             `json:"outcome"`
	Fingerprint            string             `json:"fingerprint,omitempty"`
	ContextBytes           int                `json:"context_bytes"`
	TotalContextBytes      int                `json:"total_context_bytes"`
	SelectedContext        int                `json:"selected_context"`
	OmittedContext         int                `json:"omitted_context"`
	GenerationCalled       bool               `json:"generation_called"`
	KnownGenerationUsage   bool               `json:"known_generation_usage"`
	GenerationUsage        *Usage             `json:"generation_usage,omitempty"`
	GenerationStartedAt    time.Time          `json:"generation_started_at,omitzero"`
	GenerationEndedAt      time.Time          `json:"generation_ended_at,omitzero"`
	ProviderSetupMS        float64            `json:"provider_setup_ms,omitempty"`
	ProviderSetupStartedAt time.Time          `json:"provider_setup_started_at,omitzero"`
	ProviderSetupEndedAt   time.Time          `json:"provider_setup_ended_at,omitzero"`
}

Measurement describes this invocation, never the cached answer's original latency. It is returned to the caller and is not stored in the job, included in its fingerprint, or sent anywhere automatically. Complete and durable describe protocol state, not answer quality.

type OllamaOptions added in v0.9.1 

type OllamaOptions struct {
	Format                   string            `json:"format,omitempty"`
	Temperature              *float64          `json:"temperature,omitempty"`
	KeepAliveSeconds         *int64            `json:"keep_alive_seconds,omitempty"`
	Think                    *bool             `json:"think,omitempty"`
	GenerationTimeoutSeconds *int64            `json:"generation_timeout_seconds,omitempty"`
	PromptMode               string            `json:"prompt_mode,omitempty"`
	OutputTypes              map[string]string `json:"output_types,omitempty"`
	FieldRoles               map[string]string `json:"field_roles,omitempty"`
}

OllamaOptions opts into local generation controls without changing runtime defaults for requests that omit them. Format accepts only "json" or empty; a non-nil Temperature must be finite and between 0 and 2, inclusive. KeepAliveSeconds accepts 0..300; 0 requests unloading after generation. Think defaults to false. Explicit true requires nonstream generation and installed model metadata advertising boolean true; reasoning is discarded. GenerationTimeoutSeconds accepts 1..600 for generation HTTP only; omission preserves the 120-second default. Earlier caller deadlines still apply. PromptMode accepts empty for legacy framing or "bounded_text" for the exact BoundedTextPrompt user prompt. Its request system is empty unless an explicit OutputTypes contract adds the declared type hint and effective model system. OutputTypes is an optional manually declared JSON object contract: at most 32 lower snake_case fields with primitive integer/string/number/boolean types. It requires Format=json and supplies no answer values or property constraints. The serialized decoder schema is limited to 4096 bytes; empty maps are invalid. FieldRoles optionally declares quantity, policy_action, or identifier roles for a subset of OutputTypes, supplying generic semantic context without values.

type OllamaProvider 

type OllamaProvider struct {
	// contains filtered or unexported fields
}

OllamaProvider runs installed model weights through a literal loopback HTTP endpoint. It does not sign in, pull models, send keys, or use cloud models. A provider captures one installed model snapshot; recreate it to refresh a mutable model alias after its weights or template change.

func NewOllamaProvider 

func NewOllamaProvider(baseURL, model string) (*OllamaProvider, error)

NewOllamaProvider checks that model is already installed and advertises local text completion. baseURL defaults to http://127.0.0.1:11434; literal loopback HTTP is required so a local-mode configuration cannot use a hosted endpoint or depend on DNS resolution.

func NewOllamaProviderContext 

func NewOllamaProviderContext(ctx context.Context, baseURL, model string) (*OllamaProvider, error)

NewOllamaProviderContext includes installed-model discovery in the caller's cancellation and deadline. Discovery also has a 15-second maximum of its own.

func (*OllamaProvider) Close added in v0.15.0 

func (p *OllamaProvider) Close()

Close releases idle sockets owned by this provider; it does not unload weights or cancel another inference request. The provider must no longer be in use.

func (*OllamaProvider) CountInputTokens 

func (p *OllamaProvider) CountInputTokens(ctx context.Context, req Request) (int64, error)

CountInputTokens is a conservative local policy ESTIMATE, not an Ollama tokenizer API or a tokenizer-specific proof. It allocates one token per byte of requested text and model template/system, plus 512 tokens of overhead. Metadata reads do not run inference; reported usage replaces the estimate after generation. Context and generation caps still govern local resources.

func (*OllamaProvider) Generate 

func (p *OllamaProvider) Generate(ctx context.Context, req Request) (Response, error)

func (*OllamaProvider) Identity 

func (p *OllamaProvider) Identity() string

func (*OllamaProvider) InputTokenCountBasis added in v0.13.0 

func (p *OllamaProvider) InputTokenCountBasis() string

func (*OllamaProvider) ModelDigest added in v0.3.0 

func (p *OllamaProvider) ModelDigest() string

ModelDigest exposes the installed weights identity without endpoint, prompt, template or system content. It performs no inference or metadata request.

func (*OllamaProvider) SelfHosted added in v0.10.0 

func (p *OllamaProvider) SelfHosted() bool

type OpenRouterConfig added in v0.5.0 

type OpenRouterConfig struct {
	UpstreamProvider     string `json:"upstream_provider"`
	InputTokenUpperBound int64  `json:"input_token_upper_bound"`
	CacheWriteUsage      string `json:"cache_write_usage"`
}

OpenRouterConfig describes one explicitly approved route. The input bound is supplied by the operator: OpenRouter has no documented preflight counting API for arbitrary models. It is a reservation bound, not a measured token count. CacheWriteUsage must be "required" for a route with cache-write pricing, or "not_applicable" for a route whose cache writes have no separate token charge.

type OpenRouterProvider added in v0.5.0 

type OpenRouterProvider struct {
	// contains filtered or unexported fields
}

OpenRouterProvider implements a deliberately narrow Chat Completions route: one model, one upstream, no fallbacks, retries, tools, or streaming. It does not translate OpenAI Responses cache breakpoints into gateway parameters.

func NewOpenRouterProvider added in v0.5.0 

func NewOpenRouterProvider(apiKey, baseURL string, config OpenRouterConfig) (*OpenRouterProvider, error)

func (*OpenRouterProvider) CountInputTokens added in v0.5.0 

func (p *OpenRouterProvider) CountInputTokens(ctx context.Context, req Request) (int64, error)

func (*OpenRouterProvider) Generate added in v0.5.0 

func (p *OpenRouterProvider) Generate(ctx context.Context, req Request) (Response, error)

func (*OpenRouterProvider) Identity added in v0.5.0 

func (p *OpenRouterProvider) Identity() string

func (*OpenRouterProvider) InputTokenCountBasis added in v0.13.0 

func (p *OpenRouterProvider) InputTokenCountBasis() string

CountInputTokens returns the explicit operator-approved reservation bound. No tokenizer equivalence, exact preflight count, or upstream call is implied.

type Plan 

type Plan struct {
	Fingerprint       string   `json:"fingerprint"`
	Request           Request  `json:"request"`
	Selected          []string `json:"selected_context"`
	Omitted           []string `json:"omitted_context"`
	ContextBytes      int      `json:"context_bytes"`
	TotalContextBytes int      `json:"total_context_bytes"`
}

func BuildPlan 

func BuildPlan(s Spec) (Plan, error)

BuildPlan selects whole context chunks without a model call. Required chunks must all fit. Optional chunks use explicit descending priority, then ID. The resulting prefix and input are a reviewable snapshot, never a hidden summary.

type Prices 

type Prices struct {
	Input      float64 `json:"input_per_million"`
	CacheRead  float64 `json:"cache_read_per_million"`
	CacheWrite float64 `json:"cache_write_per_million"`
	Output     float64 `json:"output_per_million"`
}

Prices are explicit USD per million tokens, not a built-in or live price list. CacheWrite must cover the cache policy in use, including any write premium.

type Provider 

type Provider interface {
	// Identity distinguishes upstream endpoints in durable request fingerprints.
	Identity() string
	CountInputTokens(context.Context, Request) (int64, error)
	Generate(context.Context, Request) (Response, error)
}

type Request 

type Request struct {
	OpenRouter      *OpenRouterConfig `json:"openrouter,omitempty"`
	SGLang          *SGLangConfig     `json:"sglang,omitempty"`
	Ollama          *OllamaOptions    `json:"ollama,omitempty"`
	Stream          bool              `json:"stream,omitempty"`
	Provider        string            `json:"provider"`
	Model           string            `json:"model"`
	ReasoningEffort string            `json:"reasoning_effort,omitempty"`
	ContextWindow   int64             `json:"context_window,omitempty"` // local runtime only
	Prefix          string            `json:"prefix"`
	Input           string            `json:"input"`
	MaxOutputTokens int64             `json:"max_output_tokens"`
	CacheMode       string            `json:"cache_mode"`
}

Request is provider-independent, single-turn text generation without tools. Prefix contains approved instructions and stable reference material; Input contains changing reference material and the user's task.

type Response 

type Response struct {
	RuntimeCache *RuntimeCacheObservation `json:"runtime_cache,omitempty"`
	Timing       *GenerationTiming        `json:"timing,omitempty"`
	ID           string                   `json:"id"`
	Model        string                   `json:"model"`
	Text         string                   `json:"text"`
	Complete     bool                     `json:"complete"`
	StopReason   string                   `json:"stop_reason"`
	Usage        Usage                    `json:"usage"`
}

type Result 

type Result struct {
	ReuseKind              string                   `json:"reuse_kind,omitempty"`
	Resolution             *ReuseResolution         `json:"resolution,omitempty"`
	BudgetDiagnostic       *BudgetDiagnostic        `json:"budget_diagnostic,omitempty"`
	GenerationRuntimeCache *RuntimeCacheObservation `json:"generation_runtime_cache,omitempty"`
	GenerationTiming       *GenerationTiming        `json:"generation_timing,omitempty"`
	Measurement            *Measurement             `json:"measurement,omitempty"`
	Fingerprint            string                   `json:"fingerprint"`
	Response               Response                 `json:"response"`
	Reused                 bool                     `json:"reused"`
	Durable                bool                     `json:"durable"`
	GenerationUsage        Usage                    `json:"generation_usage"`
	ThisRunUsage           Usage                    `json:"this_run_usage"`
	AvoidedUsage           Usage                    `json:"avoided_usage"`
	CostUSD                *float64                 `json:"cost_usd,omitempty"`
	AvoidedCostUSD         *float64                 `json:"avoided_cost_usd,omitempty"`
	CreatedAt              time.Time                `json:"created_at"`
	ExpiresAt              time.Time                `json:"expires_at"`
}

type ReuseResolution added in v0.15.0 

type ReuseResolution struct {
	PolicyID        string      `json:"policy_id"`
	Source          ReuseSource `json:"source"`
	Response        Response    `json:"response"`
	VerificationKey string      `json:"verification_key"`
}

ReuseResolution retains historical source usage in Response. It is evidence for answer reuse, not evidence of target inference or measured target savings.

type ReuseResolver added in v0.15.0 

type ReuseResolver interface {
	Identity() string
	Resolve(context.Context, Spec) (*ReuseResolution, error)
}

ReuseResolver is an explicitly authorized, bounded equivalence policy. Resolve runs only after this runner owns the target claim and before target generation. It must preserve uncertainty and accounting for any verification work it owns.

type ReuseSource added in v0.15.0 

type ReuseSource struct {
	Key                string `json:"key"`
	Version            int64  `json:"version"`
	SHA256             string `json:"sha256"`
	OutputSHA256       string `json:"output_sha256"`
	RequestFingerprint string `json:"request_fingerprint"`
}

ReuseSource pins a completed original generation. Resolved aliases cannot be sources, and a current source read must still match every supplied binding.

type Runner 

type Runner struct {
	Broker   Broker
	Provider Provider
	Now      func() time.Time
}

Runner coordinates workers via the existing broker API. Now may be set by tests; production uses the UTC wall clock. A pending claim never expires into another paid call: a crashed worker may already have incurred a charge.

func (Runner) Lookup added in v0.15.0 

func (r Runner) Lookup(ctx context.Context, s Spec) (Result, bool, error)

Lookup reads an eligible exact original generation without publishing a claim, counting tokens, reserving budget or calling Generate. Resolved aliases are a miss because this method has no policy authorizing their resolution. Pending and uncertain jobs remain errors and never turn into fresh admission.

func (Runner) Run 

func (r Runner) Run(ctx context.Context, s Spec) (result Result, runErr error)

func (Runner) RunWithResolver added in v0.15.0 

func (r Runner) RunWithResolver(ctx context.Context, s Spec, resolver ReuseResolver) (Result, error)

RunWithResolver preserves exact reuse and claim ownership, then consults the explicit resolver before any target token count, reservation or generation.

func (Runner) RuntimeIdentity added in v0.14.0 

func (r Runner) RuntimeIdentity() string

RuntimeIdentity returns a content-free binding to the provider's captured runtime identity. It performs no discovery and changes no plan or job keys.

type RuntimeCacheObservation added in v0.5.0 

type RuntimeCacheObservation struct {
	CachedTokens   *int64 `json:"cached_tokens,omitempty"`
	DeviceTokens   *int64 `json:"device_tokens,omitempty"`
	HostTokens     *int64 `json:"host_tokens,omitempty"`
	StorageTokens  *int64 `json:"storage_tokens,omitempty"`
	StorageBackend string `json:"storage_backend,omitempty"`
	Profile        string `json:"profile"`
	RuntimeVersion string `json:"runtime_version"`
	Evidence       string `json:"evidence"`
}

RuntimeCacheObservation describes compute reuse reported by a serving runtime. These counters are not billing cache buckets or proof of an invoice discount. A nil counter means unreported, rather than zero. Historical observations on a reused result describe its original generation, not a new runtime request.

type SGLangCapabilities added in v0.5.0 

type SGLangCapabilities struct {
	Profile               string `json:"profile"`
	RuntimeVersion        string `json:"runtime_version"`
	GenerationSupported   bool   `json:"generation_supported"`
	RuntimeIdentitySHA256 string `json:"runtime_identity_sha256"`
	Observed              bool   `json:"observed"`
	Reason                string `json:"reason,omitempty"`
}

SGLangCapabilities contains allowlisted readback facts, without endpoints, prompts, model paths, launch commands, or credentials. Observed configuration is not proof that a particular offload kernel ran or improved performance.

type SGLangConfig added in v0.5.0 

type SGLangConfig struct {
	DeploymentID   string `json:"deployment_id"`
	CacheNamespace string `json:"cache_namespace"`
	Profile        string `json:"profile"`
	SelfHosted     bool   `json:"self_hosted"`
	MaxInputTokens int64  `json:"max_input_tokens"`
}

SGLangConfig identifies an explicitly operated, immutable serving deployment. DeploymentID is the operator's model/configuration revision, not a weight digest proved by the HTTP API. Change it whenever that deployment changes.

type SGLangProvider added in v0.5.0 

type SGLangProvider struct {
	// contains filtered or unexported fields
}

SGLangProvider implements the pinned native single-turn, text-only contract. SGLang owns its KV tensors and storage. Questlock performs no administration, model download, runtime reconfiguration, retry, or PD bootstrap orchestration.

func NewSGLangProviderContext added in v0.5.0 

func NewSGLangProviderContext(ctx context.Context, apiKey, baseURL, model string, cfg SGLangConfig) (*SGLangProvider, error)

NewSGLangProviderContext observes an already-running SGLang v0.5.20 server. Literal loopback HTTP permits no key; remote HTTPS requires an explicit key. Constructor readbacks run no generation and include no admin writes.

func (*SGLangProvider) Capabilities added in v0.5.0 

func (p *SGLangProvider) Capabilities() SGLangCapabilities

func (*SGLangProvider) CountInputTokens added in v0.5.0 

func (p *SGLangProvider) CountInputTokens(ctx context.Context, req Request) (int64, error)

func (*SGLangProvider) Generate added in v0.5.0 

func (p *SGLangProvider) Generate(ctx context.Context, req Request) (Response, error)

func (*SGLangProvider) Identity added in v0.5.0 

func (p *SGLangProvider) Identity() string

func (*SGLangProvider) InputTokenCountBasis added in v0.13.0 

func (p *SGLangProvider) InputTokenCountBasis() string

func (*SGLangProvider) RefreshIdentity added in v0.5.0 

func (p *SGLangProvider) RefreshIdentity(ctx context.Context) error

RefreshIdentity is used before a Runner result lookup as well as around tokenization/generation. An existing provider never silently adopts a new model or restart; construct a fresh provider for that new deployment.

func (*SGLangProvider) SelfHosted added in v0.10.0 

func (p *SGLangProvider) SelfHosted() bool

type SelfHostedProvider added in v0.10.0 

type SelfHostedProvider interface {
	SelfHosted() bool
}

SelfHostedProvider is an optional assertion by a trusted provider implementation that it uses an operator-owned runtime without provider token metering. It is not remote attestation or evidence that hardware is free. Existing Provider implementations need not implement this capability.

type Spec 

type Spec struct {
	DailyTokenBudget *TokenBudget      `json:"daily_token_budget,omitempty"`
	OpenRouter       *OpenRouterConfig `json:"openrouter,omitempty"`
	SGLang           *SGLangConfig     `json:"sglang,omitempty"`
	Ollama           *OllamaOptions    `json:"ollama,omitempty"`
	Stream           bool              `json:"stream,omitempty"` // Optional local Ollama content streaming.
	Task             string            `json:"task"`
	Revision         string            `json:"revision"`
	Provider         string            `json:"provider"`
	Model            string            `json:"model"`
	ReasoningEffort  string            `json:"reasoning_effort,omitempty"`
	Instructions     string            `json:"instructions"`
	Input            string            `json:"input"`
	Context          []ContextItem     `json:"context,omitempty"`
	MaxContextBytes  int               `json:"max_context_bytes"`
	MaxInputTokens   int64             `json:"max_input_tokens"`
	MaxOutputTokens  int64             `json:"max_output_tokens"`
	CacheMode        string            `json:"cache_mode,omitempty"` // default or prefix
	ReuseForSeconds  int64             `json:"reuse_for_seconds,omitempty"`
	Prices           *Prices           `json:"prices,omitempty"`
	MaxCostUSD       float64           `json:"max_cost_usd,omitempty"`
	DailyBudgetUSD   float64           `json:"daily_budget_usd,omitempty"`
}

Spec is an explicit snapshot of a task. Reuse is opt-in and should only be enabled for tasks whose answer remains valid for the supplied revision.

type TokenBudget added in v0.10.0 

type TokenBudget struct {
	InputTokens  int64 `json:"input_tokens"`
	OutputTokens int64 `json:"output_tokens"`
	TotalTokens  int64 `json:"total_tokens"`
}

TokenBudget is a shared scope/UTC-day allowance across participating models and workers. All three caps are positive, at most 1e12, and must agree across participants. Input includes cache reads/writes; output includes reasoning. These are each model's reported token units, not equivalent units of work. A nil policy cannot bypass an existing day's ledger. Register the policy in a dedicated scope before concurrent traffic; old/arbitrary clients can bypass it.

type Usage 

type Usage struct {
	InputTokens         int64 `json:"input_tokens"`
	UncachedInputTokens int64 `json:"uncached_input_tokens"`
	CacheReadTokens     int64 `json:"cache_read_tokens"`
	CacheWriteTokens    int64 `json:"cache_write_tokens"`
	OutputTokens        int64 `json:"output_tokens"`
	ReasoningTokens     int64 `json:"reasoning_tokens,omitempty"`
}

Usage partitions input tokens into uncached, cache reads, and cache writes. InputTokens is their sum on all providers. Reasoning is already included in OutputTokens, and must not be charged a second time.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.