Documentation
¶
Index ¶
Constants ¶
const ( ServerTypeOllama = "ollama" ServerTypeLlamaServer = "llama-server" ServerTypeVLLM = "vllm" ServerTypeLMStudio = "lm-studio" ServerTypeLiteLLM = "litellm" ServerTypeOpenAICompat = "openai-compat" )
Server-type identifiers returned by DetectServerType / set as the expected label on probe entries. Centralised here so callers (internal/model/discover.go, status output) match by constant.
const LiteLLMPort = 4000
LiteLLMPort is LiteLLM's default listen port — exposed so callers like internal/model can skip it during discovery without redefining the constant.
const LocalDiscoveryPortsEnv = "OBOL_LOCAL_MODEL_DISCOVERY_PORTS"
LocalDiscoveryPortsEnv lets operators add to (or override) the default list of probed local inference ports. Format: comma-separated "port[:label]" pairs, e.g. "9000:vllm,5001:custom". Labels are informational; the actual server type is still detected by probing.
const ModeStandalone = "standalone"
ModeStandalone marks a descriptor written by a CLI where `obol sell inference` only runs as a standalone host gateway with its own x402 gate.
Variables ¶
var ErrDeploymentExists = errors.New("inference: deployment already exists")
ErrDeploymentExists is returned by Create when a deployment with the same name already exists and --force was not specified.
var ErrDeploymentNotFound = errors.New("inference: deployment not found")
ErrDeploymentNotFound is returned when a named inference deployment does not exist in the store.
var PaidRoutes = []string{
"POST /v1/chat/completions",
"POST /v1/completions",
"POST /v1/embeddings",
"GET /v1/models",
}
PaidRoutes are the "METHOD /path" patterns the gateway proxies to the upstream, each behind the x402 payment gate. Nothing else is forwarded.
Functions ¶
func FormatEndpointDisplay ¶ added in v0.8.0
func FormatEndpointDisplay(endpoints []EndpointInfo) string
FormatEndpointDisplay pretty-prints a list of discovered endpoints.
func ValidateName ¶ added in v0.5.0
ValidateName checks that name is a safe deployment identifier. It rejects empty strings, path traversal attempts, and shell metacharacters.
Types ¶
type Deployment ¶ added in v0.5.0
type Deployment struct {
// Name is the human-readable identifier for this deployment.
// Used as the directory name.
Name string `json:"name"`
// ListenAddr is the gateway listen address (default ":8402").
ListenAddr string `json:"listen_addr"`
// UpstreamURL is the inference backend URL (default "http://localhost:11434").
UpstreamURL string `json:"upstream_url"`
// WalletAddress is the USDC payment recipient.
WalletAddress string `json:"wallet_address"`
// PricePerRequest is the per-inference price denominated in AssetSymbol
// units (default "0.001" USDC).
PricePerRequest string `json:"price_per_request"`
// AssetSymbol is the token symbol charged per request (default "USDC").
// Used for human-readable log output and to label persisted state.
AssetSymbol string `json:"asset_symbol,omitempty"`
// PricePerMTok is the original per-million-token price when request pricing
// was derived from the temporary phase-1 approximation.
PricePerMTok string `json:"price_per_mtok,omitempty"`
// ApproxTokensPerRequest records the fixed approximation used to derive the
// charged request price from PricePerMTok.
ApproxTokensPerRequest int `json:"approx_tokens_per_request,omitempty"`
// Chain is the x402 payment chain name (e.g. "base", "base-sepolia").
Chain string `json:"chain"`
// FacilitatorURL is the x402 facilitator URL.
FacilitatorURL string `json:"facilitator_url"`
// Mode records how the gateway runs. Every descriptor written by the
// current CLI is ModeStandalone (built-in x402 gate, host listener).
// Descriptors without it predate the removal of cluster routing: on a
// host with a stack, resume used to publish them as cluster
// ServiceOffers fronting the host gateway. See LegacyClusterRouted.
Mode string `json:"mode,omitempty"`
// Provenance holds optional metadata about how the model was produced
// (e.g. training or benchmark results). Stored alongside the deployment
// config and passed to the registration document when selling.
Provenance *Provenance `json:"provenance,omitempty"`
// ModelName is the upstream model identifier the operator selected via
// `obol sell inference --model <name>` (e.g. "aeon-ultimate").
ModelName string `json:"model_name,omitempty"`
// ServiceNamespace is the Kubernetes namespace a legacy cluster-routed
// descriptor's Service + Endpoints + ServiceOffer live in (typically
// "llm"). Only read for legacy descriptors, so `obol sell delete` and
// the resume warning can name the right namespace.
ServiceNamespace string `json:"service_namespace,omitempty"`
// Registration is the ERC-8004 registration block the operator passed at
// sell time (--register-*). Persisted as map[string]any (the same shape
// buildSellRegistrationConfig emits) so the resume path can rebuild the
// ServiceOffer's spec.registration verbatim. Empty when the offer was
// created with --no-register.
Registration map[string]any `json:"registration,omitempty"`
// CreatedAt is the RFC3339 timestamp of when this deployment was created.
CreatedAt string `json:"created_at"`
// UpdatedAt is the RFC3339 timestamp of the most recent update.
UpdatedAt string `json:"updated_at,omitempty"`
// DeletedAt is set when `obol sell delete` removes a legacy
// cluster-routed offer from the cluster. The descriptor stays on disk
// for `sell status` history; resume skips tombstoned descriptors
// silently instead of warning about them on every stack up.
// Re-creating the offer with `obol sell inference <name>` rewrites the
// descriptor and clears the tombstone.
DeletedAt string `json:"deleted_at,omitempty"`
}
Deployment is a named, persisted inference gateway configuration. Unknown JSON fields are ignored on load, so descriptors written by older releases (e.g. with the removed "enclave_tag", "vm_*", "tee_type" or "no_payment_gate" fields) still load.
func (*Deployment) LegacyClusterRouted ¶ added in v0.17.0
func (d *Deployment) LegacyClusterRouted() bool
LegacyClusterRouted reports whether the descriptor predates standalone-only `obol sell inference`, i.e. it was (or on resume would have been) published as a cluster ServiceOffer fronting the host gateway. That routing has been removed; resume skips these descriptors and points the operator at the LiteLLM-backed `obol sell http` path instead.
type EndpointInfo ¶ added in v0.8.0
EndpointInfo describes a discovered local inference endpoint.
func ProbeEndpointContext ¶ added in v0.8.0
ProbeEndpointContext is the context-aware version of ProbeEndpoint. It creates a shared HTTP client used for both server type detection and model fetching to avoid redundant connections.
func ScanLocalEndpointsContext ¶ added in v0.8.0
func ScanLocalEndpointsContext(ctx context.Context) ([]EndpointInfo, error)
ScanLocalEndpointsContext probes common ports concurrently with context support. All ports are probed in parallel using goroutines; results are collected and returned in stable port order. The probed port list is the union of the built-in defaults and any extras from LocalDiscoveryPortsEnv.
func (EndpointInfo) BaseURL ¶ added in v0.8.0
func (e EndpointInfo) BaseURL() string
BaseURL returns the HTTP base URL for this endpoint.
type Gateway ¶
type Gateway struct {
// contains filtered or unexported fields
}
Gateway is an x402-enabled reverse proxy for LLM inference.
func NewGateway ¶
func NewGateway(cfg GatewayConfig) (*Gateway, error)
NewGateway creates a new inference gateway with the given configuration.
type GatewayConfig ¶
type GatewayConfig struct {
// ListenAddr is the address to listen on (e.g., ":8402").
ListenAddr string
// UpstreamURL is the upstream inference service URL (e.g., "http://localhost:11434").
UpstreamURL string
// WalletAddress is the USDC recipient address for payments.
WalletAddress string
// PricePerRequest is the amount charged per inference request, denominated
// in AssetSymbol units (e.g., "0.001" with AssetSymbol="USDC", or "0.023"
// with AssetSymbol="OBOL"). Atomic-unit conversion happens at the x402
// middleware boundary using the token's decimals.
PricePerRequest string
// AssetSymbol is the token symbol charged per request (default "USDC").
// Used for human-readable log output and price summaries.
AssetSymbol string
// Chain is the x402 chain configuration (e.g., x402pkg.ChainBaseMainnet).
Chain x402pkg.ChainInfo
// FacilitatorURL is the x402 facilitator service URL.
FacilitatorURL string
// VerifyOnly skips blockchain settlement after successful verification.
// Useful for testing and staging environments where no real funds are involved.
VerifyOnly bool
}
GatewayConfig holds configuration for the x402 inference gateway.
type ModelInfo ¶ added in v0.8.0
type ModelInfo struct {
ID string `json:"id"`
OwnedBy string `json:"owned_by"`
Created int64 `json:"created"`
}
ModelInfo describes a single model exposed by an inference server.
type Provenance ¶ added in v0.8.0
type Provenance struct {
Framework string `json:"framework,omitempty"` // e.g. "autoresearch"
MetricName string `json:"metricName,omitempty"` // e.g. "val_bpb"
MetricValue string `json:"metricValue,omitempty"` // e.g. "0.9973"
ExperimentID string `json:"experimentId,omitempty"` // commit hash or UUID
TrainHash string `json:"trainHash,omitempty"` // e.g. "sha256:..."
ParamCount string `json:"paramCount,omitempty"` // e.g. "50000000"
}
Provenance tracks how a model or service was produced. JSON field names use camelCase so the same document can flow through publish.py -> --provenance-file -> ServiceOffer -> agent-registration.json.
type Store ¶ added in v0.5.0
type Store struct {
// contains filtered or unexported fields
}
Store manages named inference deployment configurations on disk. Layout: <configDir>/inference/<name>/config.json
func (*Store) Create ¶ added in v0.5.0
func (s *Store) Create(d *Deployment, force bool) error
Create persists a new Deployment. Returns ErrDeploymentExists if a deployment with that name is already stored and force is false.
func (*Store) Get ¶ added in v0.5.0
func (s *Store) Get(name string) (*Deployment, error)
Get loads a Deployment by name. Returns ErrDeploymentNotFound if missing.
func (*Store) List ¶ added in v0.5.0
func (s *Store) List() ([]*Deployment, error)
List returns all deployment names in alphabetical order.
func (*Store) Update ¶ added in v0.5.0
func (s *Store) Update(d *Deployment) error
Update persists changes to an existing Deployment.