Help us shape the future of Go! Take the 2026 Go Developer Survey

inference

package
v0.17.0-rc1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 14, 2026 License: Apache-2.0 Imports: 19 Imported by: 0

Documentation

Index

Constants

View Source
const (
	ServerTypeOllama       = "ollama"
	ServerTypeLlamaServer  = "llama-server"
	ServerTypeVLLM         = "vllm"
	ServerTypeLMStudio     = "lm-studio"
	ServerTypeLiteLLM      = "litellm"
	ServerTypeOpenAICompat = "openai-compat"
)

Server-type identifiers returned by DetectServerType / set as the expected label on probe entries. Centralised here so callers (internal/model/discover.go, status output) match by constant.

View Source
const LiteLLMPort = 4000

LiteLLMPort is LiteLLM's default listen port — exposed so callers like internal/model can skip it during discovery without redefining the constant.

View Source
const LocalDiscoveryPortsEnv = "OBOL_LOCAL_MODEL_DISCOVERY_PORTS"

LocalDiscoveryPortsEnv lets operators add to (or override) the default list of probed local inference ports. Format: comma-separated "port[:label]" pairs, e.g. "9000:vllm,5001:custom". Labels are informational; the actual server type is still detected by probing.

View Source
const ModeStandalone = "standalone"

ModeStandalone marks a descriptor written by a CLI where `obol sell inference` only runs as a standalone host gateway with its own x402 gate.

Variables

View Source
var ErrDeploymentExists = errors.New("inference: deployment already exists")

ErrDeploymentExists is returned by Create when a deployment with the same name already exists and --force was not specified.

View Source
var ErrDeploymentNotFound = errors.New("inference: deployment not found")

ErrDeploymentNotFound is returned when a named inference deployment does not exist in the store.

View Source
var PaidRoutes = []string{
	"POST /v1/chat/completions",
	"POST /v1/completions",
	"POST /v1/embeddings",
	"GET /v1/models",
}

PaidRoutes are the "METHOD /path" patterns the gateway proxies to the upstream, each behind the x402 payment gate. Nothing else is forwarded.

Functions

func FormatEndpointDisplay added in v0.8.0

func FormatEndpointDisplay(endpoints []EndpointInfo) string

FormatEndpointDisplay pretty-prints a list of discovered endpoints.

func ValidateName added in v0.5.0

func ValidateName(name string) error

ValidateName checks that name is a safe deployment identifier. It rejects empty strings, path traversal attempts, and shell metacharacters.

Types

type Deployment added in v0.5.0

type Deployment struct {
	// Name is the human-readable identifier for this deployment.
	// Used as the directory name.
	Name string `json:"name"`

	// ListenAddr is the gateway listen address (default ":8402").
	ListenAddr string `json:"listen_addr"`

	// UpstreamURL is the inference backend URL (default "http://localhost:11434").
	UpstreamURL string `json:"upstream_url"`

	// WalletAddress is the USDC payment recipient.
	WalletAddress string `json:"wallet_address"`

	// PricePerRequest is the per-inference price denominated in AssetSymbol
	// units (default "0.001" USDC).
	PricePerRequest string `json:"price_per_request"`

	// AssetSymbol is the token symbol charged per request (default "USDC").
	// Used for human-readable log output and to label persisted state.
	AssetSymbol string `json:"asset_symbol,omitempty"`

	// PricePerMTok is the original per-million-token price when request pricing
	// was derived from the temporary phase-1 approximation.
	PricePerMTok string `json:"price_per_mtok,omitempty"`

	// ApproxTokensPerRequest records the fixed approximation used to derive the
	// charged request price from PricePerMTok.
	ApproxTokensPerRequest int `json:"approx_tokens_per_request,omitempty"`

	// Chain is the x402 payment chain name (e.g. "base", "base-sepolia").
	Chain string `json:"chain"`

	// FacilitatorURL is the x402 facilitator URL.
	FacilitatorURL string `json:"facilitator_url"`

	// Mode records how the gateway runs. Every descriptor written by the
	// current CLI is ModeStandalone (built-in x402 gate, host listener).
	// Descriptors without it predate the removal of cluster routing: on a
	// host with a stack, resume used to publish them as cluster
	// ServiceOffers fronting the host gateway. See LegacyClusterRouted.
	Mode string `json:"mode,omitempty"`

	// Provenance holds optional metadata about how the model was produced
	// (e.g. training or benchmark results). Stored alongside the deployment
	// config and passed to the registration document when selling.
	Provenance *Provenance `json:"provenance,omitempty"`

	// ModelName is the upstream model identifier the operator selected via
	// `obol sell inference --model <name>` (e.g. "aeon-ultimate").
	ModelName string `json:"model_name,omitempty"`

	// ServiceNamespace is the Kubernetes namespace a legacy cluster-routed
	// descriptor's Service + Endpoints + ServiceOffer live in (typically
	// "llm"). Only read for legacy descriptors, so `obol sell delete` and
	// the resume warning can name the right namespace.
	ServiceNamespace string `json:"service_namespace,omitempty"`

	// Registration is the ERC-8004 registration block the operator passed at
	// sell time (--register-*). Persisted as map[string]any (the same shape
	// buildSellRegistrationConfig emits) so the resume path can rebuild the
	// ServiceOffer's spec.registration verbatim. Empty when the offer was
	// created with --no-register.
	Registration map[string]any `json:"registration,omitempty"`

	// CreatedAt is the RFC3339 timestamp of when this deployment was created.
	CreatedAt string `json:"created_at"`

	// UpdatedAt is the RFC3339 timestamp of the most recent update.
	UpdatedAt string `json:"updated_at,omitempty"`

	// DeletedAt is set when `obol sell delete` removes a legacy
	// cluster-routed offer from the cluster. The descriptor stays on disk
	// for `sell status` history; resume skips tombstoned descriptors
	// silently instead of warning about them on every stack up.
	// Re-creating the offer with `obol sell inference <name>` rewrites the
	// descriptor and clears the tombstone.
	DeletedAt string `json:"deleted_at,omitempty"`
}

Deployment is a named, persisted inference gateway configuration. Unknown JSON fields are ignored on load, so descriptors written by older releases (e.g. with the removed "enclave_tag", "vm_*", "tee_type" or "no_payment_gate" fields) still load.

func (*Deployment) LegacyClusterRouted added in v0.17.0

func (d *Deployment) LegacyClusterRouted() bool

LegacyClusterRouted reports whether the descriptor predates standalone-only `obol sell inference`, i.e. it was (or on resume would have been) published as a cluster ServiceOffer fronting the host gateway. That routing has been removed; resume skips these descriptors and points the operator at the LiteLLM-backed `obol sell http` path instead.

type EndpointInfo added in v0.8.0

type EndpointInfo struct {
	Host       string
	Port       int
	ServerType string
	Models     []ModelInfo
	Healthy    bool
}

EndpointInfo describes a discovered local inference endpoint.

func ProbeEndpointContext added in v0.8.0

func ProbeEndpointContext(ctx context.Context, host string, port int) (*EndpointInfo, error)

ProbeEndpointContext is the context-aware version of ProbeEndpoint. It creates a shared HTTP client used for both server type detection and model fetching to avoid redundant connections.

func ScanLocalEndpointsContext added in v0.8.0

func ScanLocalEndpointsContext(ctx context.Context) ([]EndpointInfo, error)

ScanLocalEndpointsContext probes common ports concurrently with context support. All ports are probed in parallel using goroutines; results are collected and returned in stable port order. The probed port list is the union of the built-in defaults and any extras from LocalDiscoveryPortsEnv.

func (EndpointInfo) BaseURL added in v0.8.0

func (e EndpointInfo) BaseURL() string

BaseURL returns the HTTP base URL for this endpoint.

type Gateway

type Gateway struct {
	// contains filtered or unexported fields
}

Gateway is an x402-enabled reverse proxy for LLM inference.

func NewGateway

func NewGateway(cfg GatewayConfig) (*Gateway, error)

NewGateway creates a new inference gateway with the given configuration.

func (*Gateway) Start

func (g *Gateway) Start() error

Start begins serving the gateway. Blocks until the server is shut down.

func (*Gateway) Stop

func (g *Gateway) Stop() error

Stop gracefully shuts down the gateway.

type GatewayConfig

type GatewayConfig struct {
	// ListenAddr is the address to listen on (e.g., ":8402").
	ListenAddr string

	// UpstreamURL is the upstream inference service URL (e.g., "http://localhost:11434").
	UpstreamURL string

	// WalletAddress is the USDC recipient address for payments.
	WalletAddress string

	// PricePerRequest is the amount charged per inference request, denominated
	// in AssetSymbol units (e.g., "0.001" with AssetSymbol="USDC", or "0.023"
	// with AssetSymbol="OBOL"). Atomic-unit conversion happens at the x402
	// middleware boundary using the token's decimals.
	PricePerRequest string

	// AssetSymbol is the token symbol charged per request (default "USDC").
	// Used for human-readable log output and price summaries.
	AssetSymbol string

	// Chain is the x402 chain configuration (e.g., x402pkg.ChainBaseMainnet).
	Chain x402pkg.ChainInfo

	// FacilitatorURL is the x402 facilitator service URL.
	FacilitatorURL string

	// VerifyOnly skips blockchain settlement after successful verification.
	// Useful for testing and staging environments where no real funds are involved.
	VerifyOnly bool
}

GatewayConfig holds configuration for the x402 inference gateway.

type ModelInfo added in v0.8.0

type ModelInfo struct {
	ID      string `json:"id"`
	OwnedBy string `json:"owned_by"`
	Created int64  `json:"created"`
}

ModelInfo describes a single model exposed by an inference server.

type Provenance added in v0.8.0

type Provenance struct {
	Framework    string `json:"framework,omitempty"`    // e.g. "autoresearch"
	MetricName   string `json:"metricName,omitempty"`   // e.g. "val_bpb"
	MetricValue  string `json:"metricValue,omitempty"`  // e.g. "0.9973"
	ExperimentID string `json:"experimentId,omitempty"` // commit hash or UUID
	TrainHash    string `json:"trainHash,omitempty"`    // e.g. "sha256:..."
	ParamCount   string `json:"paramCount,omitempty"`   // e.g. "50000000"
}

Provenance tracks how a model or service was produced. JSON field names use camelCase so the same document can flow through publish.py -> --provenance-file -> ServiceOffer -> agent-registration.json.

type Store added in v0.5.0

type Store struct {
	// contains filtered or unexported fields
}

Store manages named inference deployment configurations on disk. Layout: <configDir>/inference/<name>/config.json

func NewStore added in v0.5.0

func NewStore(configDir string) *Store

NewStore returns a Store rooted at configDir.

func (*Store) Create added in v0.5.0

func (s *Store) Create(d *Deployment, force bool) error

Create persists a new Deployment. Returns ErrDeploymentExists if a deployment with that name is already stored and force is false.

func (*Store) Get added in v0.5.0

func (s *Store) Get(name string) (*Deployment, error)

Get loads a Deployment by name. Returns ErrDeploymentNotFound if missing.

func (*Store) List added in v0.5.0

func (s *Store) List() ([]*Deployment, error)

List returns all deployment names in alphabetical order.

func (*Store) Update added in v0.5.0

func (s *Store) Update(d *Deployment) error

Update persists changes to an existing Deployment.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL