Help us shape the future of Go! Take the 2026 Go Developer Survey

llm

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 2, 2026 License: Apache-2.0 Imports: 32 Imported by: 0

Documentation

Overview

Schema and loader for the _experts.yml manifest.

The ExpertsManifest struct owns the `experts:` block. Consumers that need additional top-level keys (jumalu's `graphs:`) embed this struct and add their own fields; yaml.v3 inlines anonymous embedded structs, so a single YAML file is decoded into the merged schema without a custom unmarshaler.

Shared JSON POST transport for every non-streaming LLM provider call. Marshal, POST, status classification, body read, unmarshal — one place.

Providers supply URL, headers, body, client, and codec. They do not repeat the boilerplate. Streaming stays in sse.go and the provider Stream methods; this helper is for the request/response cycle only.

ai/llm/observe.go

Ollama does not use SSE. Its /api/chat stream is newline-delimited JSON, terminated by a `done:true` field inside the last line. Its streaming reader therefore stays local to this file and is not routed through StreamSSE.

SSE parsing skeleton. The four streaming providers (OpenAI-compatible, DeepSeek-via-OpenAI, Anthropic, Gemini) differ only in the JSON shape of a single `data:` frame. The reader loop, the [DONE] terminator, the context check, and the close-on-exit all live here.

Ollama does not use this file — it streams newline-delimited JSON, not SSE. Its reader stays local to ollama.go.

Index

Constants

View Source 
const DefaultPricingTemplate = `` /* 127-byte string literal not displayed */

DefaultPricingTemplate intentionally has no provider/model IDs, rates, currency, or billing window. Applications must load those from their operator-managed rate-card configuration.

Variables

View Source 
var GlobalCatalog = NewPriceCatalog()

GlobalCatalog is the default catalog used by CalculateCost and by every provider that builds a Response.

View Source 
var Presets = map[string]ProviderPreset{

	"groq": {
		Name:        "groq",
		DefaultURL:  "https://api.groq.com/openai/v1",
		EnvKeyNames: []string{"GROQ_API_KEY"},
	},

	"nim": {
		Name:        "nim",
		DefaultURL:  "http://localhost:8000/v1",
		EnvKeyNames: []string{"NVIDIA_API_KEY", "NVIDIA_NIM_API_KEY"},
	},

	"vllm": {
		Name:        "vllm",
		DefaultURL:  "http://localhost:8000/v1",
		EnvKeyNames: []string{"VLLM_API_KEY"},
	},
	// contains filtered or unexported fields
}

Presets contain provider endpoints and credential environment names only. Model identities are selected by the application or supplied per request.

Functions

func BuildMoEAdminActions 

func BuildMoEAdminActions(cfg MoEAdminConfig) []action.AnyAction

func BuildMoEExperts 

func BuildMoEExperts(manifest *ExpertsManifest) (map[string]Provider, error)

BuildMoEExperts constructs the named Provider map from a manifest.

Side effect: registers a rate card into GlobalCatalog only when the expert explicitly supplies its rate_card_id, currency, and rates. Provider/model identities and rates are external configuration; this function does not infer a card identity, currency, or price.

When an expert declares more than one provider, the returned provider is a ProviderChain composed in declaration order. The chain inherits the default fallback policy (xerr.IsTransient) and no observer.

The graph compilation layer (jumalu) is expected to walk its own `graphs:` block after this call returns. This function does not compile or execute anything.

func CalculateCost 

func CalculateCost(
	model string,
	promptTokens, cachedTokens, compTokens int,
	at time.Time,
) (cost.Micro, cost.Currency, bool)

CalculateCost returns the cost of one LLM turn. 

model       the model identifier as reported by the provider
promptTokens total input tokens (cached + miss)
cachedTokens the portion of promptTokens that hit the prompt cache.
             Pass 0 when the provider does not report cache hits.
compTokens  output tokens
at          the time of the call, used to select peak vs off-peak

Returns (micros, currency, ok). When ok is false the model has no rate card — the caller should treat cost as unknown, not zero.

func CalculateCostForRateCard added in v0.2.0 

func CalculateCostForRateCard(
	model, rateCardID string,
	expectedCurrency cost.Currency,
	promptTokens, cachedTokens, compTokens int,
	at time.Time,
) (cost.Micro, cost.Currency, bool)

CalculateCostForRateCard estimates cost only when the exact configured rate-card identity and expected currency both match. A missing card, currency, or currency mismatch returns known=false and never a zero-cost success. Explicit-card estimates round up to the next micro-unit so a budget preflight cannot pass on a sub-micro rounding underestimate.

func DecodeAnthropicChunk 

func DecodeAnthropicChunk(data []byte) (text string, done bool, err error)

DecodeAnthropicChunk handles content_block_delta and message_stop.

func DecodeGeminiChunk 

func DecodeGeminiChunk(data []byte) (text string, done bool, err error)

DecodeGeminiChunk handles the candidates/parts/text shape. Multiple parts in a single frame are concatenated; the caller sees one fragment per frame.

func DecodeOpenAIChunk 

func DecodeOpenAIChunk(data []byte) (text string, done bool, err error)

DecodeOpenAIChunk handles the OpenAI / DeepSeek / Groq / most OpenAI-compatible SSE delta shape.

func DoJSON 

func DoJSON[Res any](ctx context.Context, r HTTPRequest) (Res, error)

DoJSON performs a JSON POST, decodes the JSON response into Res, and maps non-2xx status codes through FromHTTPStatus. The function is the single HTTP entry point for every OpenAI-compatible, Anthropic, and Ollama non-streaming request.

func FetchModels 

func FetchModels(ctx context.Context, providerType, apiKey, baseURL string) ([]string, error)

FetchModels queries an upstream provider for its active model catalog. It returns no fabricated model IDs when the provider catalog is unavailable.

func FromHTTPStatus 

func FromHTTPStatus(status int, body []byte, providerName string) error

FromHTTPStatus maps an upstream HTTP response into the kernel error taxonomy. Only documented transient statuses or exact structured error codes are retryable; arbitrary body text and malformed responses are not.

func FromStreamHTTPStatus added in v0.2.0 

func FromStreamHTTPStatus(status int, body []byte, providerName string) error

FromStreamHTTPStatus applies the provider classifier to stream startup failures. A successful but non-200 response is still a terminal protocol defect for a streaming endpoint, not a successful empty stream.

func InitPricingFile 

func InitPricingFile(filePath string, force bool) (string, error)

InitPricingFile writes the default template to disk.

func IsKnownProvider 

func IsKnownProvider(name string) bool

IsKnownProvider reports whether a provider name is present in the preset catalog. It is used by the @llm directive to reject misspelled provider names at compile time.

func IsLoaded 

func IsLoaded(ctx context.Context, p Provider, model string) bool

IsLoaded checks whether the model is actively resident in VRAM.

func KnownProviders 

func KnownProviders() []string

KnownProviders returns every provider name accepted by IsKnownProvider, sorted for stable error messages.

func NewNode 

func NewNode(id, _ string, p Provider, systemPrompt, userPromptTmpl string) action.AnyAction

func NewStructuredNode 

func NewStructuredNode[T any](id, _ string, p Provider, systemPrompt, userPromptTmpl string) action.AnyAction

func NewTypedNode 

func NewTypedNode[Res any](
	name string,
	provider Provider,
	defaultSystem string,
) action.AnyAction

NewTypedNode returns an action that:

  1. Sends {system, user} to the provider with JSONMode enabled.
  2. Decodes the model's reply into Res.
  3. Returns Res as the action output.

This is the single primitive that lets every ai/* subsystem drop its own hand-rolled "prompt → call → unmarshal" boilerplate and keep only the DTO. The heavy lifting (structured output enforcement, JSON decoding, error normalisation) lives in exactly one place.

The name is required and becomes the action's registry identity (e.g. "ai.structured.review_report"). Callers typically pass a fully qualified name so the .nflow file can reference it directly.

func ParseOpenAISSEStream 

func ParseOpenAISSEStream(ctx context.Context, body io.ReadCloser) iter.Seq2[string, error]

ParseOpenAISSEStream parses an OpenAI-compatible SSE body into a standard iter.Seq2 iterator.

func Preload 

func Preload(ctx context.Context, p Provider, model string) error

Preload guarantees the model is loaded in memory/VRAM and ready for traffic.

func ProviderAction 

func ProviderAction(name string, p Provider) *action.Builder[Request, Response]

func ResolveConfigPath 

func ResolveConfigPath(flag, envKey string, defaults ...string) (string, error)

ResolveConfigPath applies the standard fail-fast resolution chain: flag wins, then env, then the first existing default. An explicit override that points to a missing file is a hard error.

func StreamSSE 

func StreamSSE(ctx context.Context, body io.ReadCloser, decode SSEChunkDecoder) iter.Seq2[string, error]

StreamSSE reads an SSE body line by line and calls decode on every `data:` payload. The function owns the reader lifecycle: it closes body on exit, checks ctx on every iteration, and stops on [DONE] or on the first decoder that returns done=true.

func Unload 

func Unload(ctx context.Context, p Provider, model string) error

Unload releases model resources from VRAM if supported.

Types

type AnthropicProvider 

type AnthropicProvider struct {
	// contains filtered or unexported fields
}

func NewAnthropic 

func NewAnthropic(cfg Config) *AnthropicProvider

func (*AnthropicProvider) Complete 

func (p *AnthropicProvider) Complete(ctx context.Context, req Request) (Response, error)

func (*AnthropicProvider) Name 

func (p *AnthropicProvider) Name() string

func (*AnthropicProvider) Stream 

func (p *AnthropicProvider) Stream(ctx context.Context, req Request) (iter.Seq2[string, error], error)

type Config 

type Config struct {
	Name         string        `json:"name,omitempty" yaml:"name,omitempty"`
	BaseURL      string        `json:"base_url,omitempty" yaml:"base_url,omitempty"`
	APIKey       string        `json:"api_key,omitempty" yaml:"api_key,omitempty"`
	DefaultModel string        `json:"default_model,omitempty" yaml:"default_model,omitempty"`
	Timeout      time.Duration `json:"timeout,omitempty" yaml:"timeout,omitempty"`
	MaxRetries   int           `json:"max_retries,omitempty" yaml:"max_retries,omitempty"`
}

type DecisionAnswer added in v0.2.0 

type DecisionAnswer struct {
	ID            string                `json:"id"`
	Type          DecisionQuestionType  `json:"type"`
	Noul          *float64              `json:"noul,omitempty"`
	Choice        string                `json:"choice,omitempty"`
	Score         *float64              `json:"score,omitempty"`
	Probabilities []DecisionProbability `json:"probabilities,omitempty"`
	Confidence    *float64              `json:"confidence,omitempty"`
	Legend        []DecisionCriterion   `json:"legend,omitempty"`
	Extractions   []DecisionExtraction  `json:"extractions,omitempty"`
}

DecisionAnswer is one typed answer, returned in the same order as the questions in DecisionRequest.

type DecisionCriterion added in v0.2.0 

type DecisionCriterion struct {
	ID          string `json:"id"`
	Description string `json:"description"`
}

DecisionCriterion names one candidate or one ordered score level. The provider adapter translates the slice into the wire format required by its API; callers never need to construct an untyped JSON object.

type DecisionExtraction added in v0.2.0 

type DecisionExtraction struct {
	Field                string         `json:"field"`
	Spans                []DecisionSpan `json:"spans"`
	NotStatedProbability float64        `json:"not_stated_probability"`
}

DecisionExtraction contains exact spans and the probability that a field was not present in the source text.

type DecisionProbability added in v0.2.0 

type DecisionProbability struct {
	ID          string  `json:"id"`
	Probability float64 `json:"probability"`
}

DecisionProbability is one named outcome probability returned by a provider.

type DecisionProvider added in v0.2.0 

type DecisionProvider interface {
	Name() string
	Decide(ctx context.Context, req DecisionRequest) (DecisionResponse, error)
}

DecisionProvider exposes structured decisions independently from chat completions. This matters for providers whose decision endpoint is not a chat-completions endpoint.

type DecisionQuestion added in v0.2.0 

type DecisionQuestion struct {
	ID               string               `json:"id"`
	Type             DecisionQuestionType `json:"type"`
	Instructions     string               `json:"instructions,omitempty"`
	Criteria         []DecisionCriterion  `json:"criteria,omitempty"`
	TrueDescription  string               `json:"true,omitempty"`
	FalseDescription string               `json:"false,omitempty"`
	Source           string               `json:"source,omitempty"`
	Threshold        *float64             `json:"threshold,omitempty"`
	SnapToWords      *bool                `json:"snap_to_words,omitempty"`
}

DecisionQuestion describes one named question evaluated against shared state. Criteria are ordered for score questions and keyed by ID for choice and extract questions.

type DecisionQuestionType added in v0.2.0 

type DecisionQuestionType string

DecisionQuestionType identifies a structured classification task supported by a decision-capable provider. 

const (
	DecisionNoul    DecisionQuestionType = "noul"
	DecisionChoice  DecisionQuestionType = "choice"
	DecisionScore   DecisionQuestionType = "score"
	DecisionExtract DecisionQuestionType = "extract"
)

type DecisionRequest added in v0.2.0 

type DecisionRequest struct {
	Model                       string             `json:"model"`
	AttemptID                   string             `json:"attempt_id,omitempty"`
	RateCardID                  string             `json:"rate_card_id,omitempty"`
	ExpectedCurrency            *cost.Currency     `json:"expected_currency,omitempty"`
	EstimatedPromptTokens       int                `json:"estimated_prompt_tokens,omitempty"`
	EstimatedCachedPromptTokens int                `json:"estimated_cached_prompt_tokens,omitempty"`
	EstimatedCompletionTokens   int                `json:"estimated_completion_tokens,omitempty"`
	MaxOutputTokens             int                `json:"max_output_tokens"`
	State                       json.RawMessage    `json:"state"`
	Questions                   []DecisionQuestion `json:"questions"`
	AllowedIDs                  []string           `json:"allowed_ids,omitempty"`
}

DecisionRequest is the typed provider request. MaxOutputTokens is a required positive hard cap; estimated token fields remain host-side accounting metadata. State is JSON so applications can pass context without a domain schema.

type DecisionResponse added in v0.2.0 

type DecisionResponse struct {
	Model   string           `json:"model"`
	Answers []DecisionAnswer `json:"answers"`
	Usage   Usage            `json:"usage"`
}

DecisionResponse is a structured result from a decision-capable model.

type DecisionSpan added in v0.2.0 

type DecisionSpan struct {
	Text       string  `json:"text"`
	Start      int     `json:"start"`
	End        int     `json:"end"`
	Confidence float64 `json:"confidence"`
}

DecisionSpan is an exact source substring found by an extraction question.

type DeepSeekProvider 

type DeepSeekProvider struct {
	// contains filtered or unexported fields
}

func NewDeepSeek 

func NewDeepSeek(cfg Config) *DeepSeekProvider

func (*DeepSeekProvider) Complete 

func (p *DeepSeekProvider) Complete(ctx context.Context, req Request) (Response, error)

func (*DeepSeekProvider) Name 

func (p *DeepSeekProvider) Name() string

func (*DeepSeekProvider) Stream 

func (p *DeepSeekProvider) Stream(ctx context.Context, req Request) (iter.Seq2[string, error], error)

type Embedder 

type Embedder interface {
	Embed(ctx context.Context, texts ...string) ([][]float32, error)
}

func NewEmbedder 

func NewEmbedder(baseURL, apiKey, model string) Embedder

NewEmbedder creates an embedding generator compatible with OpenAI, OpenRouter, and Ollama.

type ExpertProviderConfig 

type ExpertProviderConfig struct {
	Provider              string        `json:"provider" yaml:"provider"`
	Model                 string        `json:"model" yaml:"model"`
	RateCardID            string        `json:"rate_card_id,omitempty" yaml:"rate_card_id,omitempty"`
	Currency              cost.Currency `json:"currency,omitempty" yaml:"currency,omitempty"`
	Temperature           float64       `json:"temperature,omitempty" yaml:"temperature,omitempty"`
	MaxTokens             int           `json:"max_tokens,omitempty" yaml:"max_tokens,omitempty"`
	TimeoutMS             int64         `json:"timeout_ms,omitempty" yaml:"timeout_ms,omitempty"`
	SystemPrefix          string        `json:"system_prefix,omitempty" yaml:"system_prefix,omitempty"`
	PromptPrice1k         float64       `json:"prompt_price_1k,omitempty" yaml:"prompt_price_1k,omitempty"`
	PromptCacheHitPrice1k float64       `json:"prompt_cache_hit_price_1k,omitempty" yaml:"prompt_cache_hit_price_1k,omitempty"`
	CompletionPrice1k     float64       `json:"completion_price_1k,omitempty" yaml:"completion_price_1k,omitempty"`
}

ExpertProviderConfig is the per-provider entry under an expert.

TimeoutMS accepts both the current `timeout_ms:` YAML key and the historical `timeoutms:` key. The legacy form exists because an earlier loader carried the field with no yaml tag, making the literal lowercased Go field name the effective YAML key. Both forms are decoded; the canonical form wins if both are present.

func (*ExpertProviderConfig) UnmarshalYAML 

func (c *ExpertProviderConfig) UnmarshalYAML(value *yaml.Node) error

UnmarshalYAML decodes both `timeout_ms` and the legacy `timeoutms` key. Everything else uses the default yaml.v3 behavior via a shadow struct.

type ExpertSpec 

type ExpertSpec struct {
	Key         string                 `json:"key" yaml:"key"`
	Name        string                 `json:"name" yaml:"name"`
	Description string                 `json:"description" yaml:"description"`
	Keywords    []string               `json:"keywords" yaml:"keywords"`
	Providers   []ExpertProviderConfig `json:"providers" yaml:"providers"`
}

ExpertSpec is a single named expert entry.

type ExpertsManifest 

type ExpertsManifest struct {
	Experts map[string]ExpertSpec `json:"experts" yaml:"experts"`
}

ExpertsManifest is the shared top-level shape of an _experts.yml.

Consumers that need additional top-level keys should embed this struct anonymously; yaml.v3 will inline the embedded fields.

func LoadExpertsFile 

func LoadExpertsFile(filePath string) (*ExpertsManifest, error)

func LoadExpertsManifest 

func LoadExpertsManifest(data []byte) (*ExpertsManifest, error)

func (*ExpertsManifest) MatchExpertByKeywords 

func (m *ExpertsManifest) MatchExpertByKeywords(query string) string

MatchExpertByKeywords returns the first expert whose keyword list contains a case-insensitive substring of query. Returns "general" when nothing matches or the manifest is empty.

type FallbackConfig 

type FallbackConfig struct {
	ShouldFallback func(error) bool
	Transform      func(Request) Request
	OnFallback     func(context.Context, FallbackInfo) error
}

FallbackConfig controls provider failover conditions and request transforms.

type FallbackError 

type FallbackError struct {
	Primary   error
	Secondary error
}

FallbackError wraps primary and secondary errors for errors.Is / errors.As matching.

func (*FallbackError) Error 

func (e *FallbackError) Error() string

func (*FallbackError) Unwrap 

func (e *FallbackError) Unwrap() []error

type FallbackInfo 

type FallbackInfo struct {
	PrimaryRequest  Request
	FallbackRequest Request
	PrimaryError    error
}

FallbackInfo captures primary-to-secondary provider transition context.

type FallbackProvider 

type FallbackProvider struct {
	// contains filtered or unexported fields
}

FallbackProvider executes secondary provider on transient primary failure.

func NewFallbackProvider 

func NewFallbackProvider(primary, secondary Provider, cfg FallbackConfig) *FallbackProvider

func (*FallbackProvider) Complete 

func (p *FallbackProvider) Complete(ctx context.Context, req Request) (Response, error)

func (*FallbackProvider) Name 

func (p *FallbackProvider) Name() string

func (*FallbackProvider) Stream 

func (p *FallbackProvider) Stream(ctx context.Context, req Request) (iter.Seq2[string, error], error)

type FunctionCall 

type FunctionCall struct {
	Name      string `json:"name"`
	Arguments string `json:"arguments"`
}

type FunctionSpec 

type FunctionSpec struct {
	Name        string `json:"name"`
	Description string `json:"description"`
	Parameters  any    `json:"parameters"`
	Example     any    `json:"example,omitempty"`
}

type GeminiProvider 

type GeminiProvider struct {
	// contains filtered or unexported fields
}

func NewGemini 

func NewGemini(cfg Config) *GeminiProvider

func (*GeminiProvider) Complete 

func (p *GeminiProvider) Complete(ctx context.Context, req Request) (Response, error)

func (*GeminiProvider) Name 

func (p *GeminiProvider) Name() string

func (*GeminiProvider) Stream 

func (p *GeminiProvider) Stream(ctx context.Context, req Request) (iter.Seq2[string, error], error)

type HTTPRequest 

type HTTPRequest struct {
	URL          string
	Headers      map[string]string
	Body         any
	Client       *http.Client
	Codec        codec.Codec
	ProviderName string
}

HTTPRequest is the input to DoJSON. ProviderName is used only in error messages produced by FromHTTPStatus; leave empty to omit the prefix.

type InceptionProvider added in v0.2.0 

type InceptionProvider struct {
	// contains filtered or unexported fields
}

InceptionProvider supports OpenAI-compatible chat completions and exposes the native decision interface. The documented /decisions API has no provider-enforced output-token cap, so Decide refuses dispatch rather than silently ignoring DecisionRequest.MaxOutputTokens. Use the cap-aware inception-chat adapter for structured JSON decisions instead.

func NewInception added in v0.2.0 

func NewInception(cfg Config) *InceptionProvider

func (*InceptionProvider) Complete added in v0.2.0 

func (p *InceptionProvider) Complete(ctx context.Context, req Request) (Response, error)

func (*InceptionProvider) Decide added in v0.2.0 

func (p *InceptionProvider) Decide(_ context.Context, req DecisionRequest) (DecisionResponse, error)

func (*InceptionProvider) Name added in v0.2.0 

func (p *InceptionProvider) Name() string

func (*InceptionProvider) Stream added in v0.2.0 

func (p *InceptionProvider) Stream(ctx context.Context, req Request) (iter.Seq2[string, error], error)

type JSONDecisionProvider added in v0.2.0 

type JSONDecisionProvider struct {
	// contains filtered or unexported fields
}

JSONDecisionProvider adapts a chat provider with JSON mode into the typed DecisionProvider interface. Provider outputs are parsed and validated locally; unlike Inception's native adapter, this does not use a vendor decision endpoint.

func NewAnthropicDecisionProvider added in v0.2.0 

func NewAnthropicDecisionProvider(provider *AnthropicProvider) *JSONDecisionProvider

NewAnthropicDecisionProvider enables Anthropic Messages' native JSON Schema output. Other providers should use NewJSONDecisionProvider unless their provider-specific adapter is implemented separately.

func NewJSONDecisionProvider added in v0.2.0 

func NewJSONDecisionProvider(provider Provider) *JSONDecisionProvider

func (*JSONDecisionProvider) Decide added in v0.2.0 

func (p *JSONDecisionProvider) Decide(ctx context.Context, req DecisionRequest) (DecisionResponse, error)

func (*JSONDecisionProvider) Name added in v0.2.0 

func (p *JSONDecisionProvider) Name() string

type JSONSchema added in v0.2.0 

type JSONSchema struct {
	Type                 string                `json:"type"`
	Properties           map[string]JSONSchema `json:"properties,omitempty"`
	Required             []string              `json:"required,omitempty"`
	Items                *JSONSchema           `json:"items,omitempty"`
	Enum                 []string              `json:"enum,omitempty"`
	AdditionalProperties *bool                 `json:"additionalProperties,omitempty"`
}

JSONSchema is the typed subset of JSON Schema used for provider-constrained structured outputs. Provider adapters translate it to their native API shape.

type Message 

type Message struct {
	Role             Role       `json:"role"`
	Content          string     `json:"content"`
	ReasoningContent string     `json:"reasoning_content,omitempty"`
	Name             string     `json:"name,omitempty"`
	ToolCallID       string     `json:"tool_call_id,omitempty"`
	ToolCalls        []ToolCall `json:"tool_calls,omitempty"`
}

type MoEAdminConfig 

type MoEAdminConfig struct {
	Sink           *RouterSink
	ReloadFunc     func(context.Context) (int, error)
	KnownProviders []string
}

MoEAdminConfig wires the MoE admin surface to a RouterSink and a manifest reload callback supplied by the caller.

type MoEModelsReq 

type MoEModelsReq struct {
	Provider string `json:"provider" validate:"required"`
	APIKey   string `json:"api_key,omitempty"`
	BaseURL  string `json:"base_url,omitempty"`
}

type MoEModelsRes 

type MoEModelsRes struct {
	Provider string   `json:"provider"`
	Models   []string `json:"models"`
}

type MoEProvider 

type MoEProvider struct {
	// contains filtered or unexported fields
}

func BuildMoEFromManifest 

func BuildMoEFromManifest(manifest *ExpertsManifest, defaultProvider Provider) (*MoEProvider, error)

BuildMoEFromManifest is the legacy entry point. It wraps the expert map into a MoEProvider for callers that want a single routing provider.

New callers should prefer BuildMoEExperts and construct their own router (TieredModelRouter, ProviderChain, or a custom dispatcher).

func NewMoE 

func NewMoE(name string, defaultProvider Provider) *MoEProvider

func (*MoEProvider) Complete 

func (m *MoEProvider) Complete(ctx context.Context, req Request) (Response, error)

func (*MoEProvider) IsLoaded 

func (m *MoEProvider) IsLoaded(ctx context.Context, _ string) bool

func (*MoEProvider) Name 

func (m *MoEProvider) Name() string

func (*MoEProvider) Preload 

func (m *MoEProvider) Preload(ctx context.Context, _ string) error

func (*MoEProvider) RegisterExpert 

func (m *MoEProvider) RegisterExpert(expertName string, p Provider) *MoEProvider

func (*MoEProvider) Stream 

func (m *MoEProvider) Stream(ctx context.Context, req Request) (iter.Seq2[string, error], error)

func (*MoEProvider) Unload 

func (m *MoEProvider) Unload(ctx context.Context, _ string) error

type MoEReloadRes 

type MoEReloadRes struct {
	Experts int `json:"experts_reloaded"`
}

type MoEStatsRes 

type MoEStatsRes struct {
	Executed uint64 `json:"executed"`
	Retries  uint64 `json:"retries"`
	Errors   uint64 `json:"errors"`
}

type ModelManager 

type ModelManager interface {
	Preload(ctx context.Context, model string) error
	Unload(ctx context.Context, model string) error
	IsLoaded(ctx context.Context, model string) bool
}

type ObservableProvider 

type ObservableProvider struct {
	// contains filtered or unexported fields
}

ObservableProvider wraps any Provider and fires a ProviderTrace around every Complete call. Stream is not traced; use the runner's stream events for streaming observability.

func (*ObservableProvider) Complete 

func (p *ObservableProvider) Complete(ctx context.Context, req Request) (Response, error)

func (*ObservableProvider) Name 

func (p *ObservableProvider) Name() string

func (*ObservableProvider) Stream 

func (p *ObservableProvider) Stream(ctx context.Context, req Request) (iter.Seq2[string, error], error)

type OffPeakWindow 

type OffPeakWindow struct {
	StartUTC string `json:"start_utc" yaml:"start_utc"`
	EndUTC   string `json:"end_utc"   yaml:"end_utc"`
}

OffPeakWindow is the daily UTC window during which the OffPeak rate card applies. The window may wrap midnight (e.g. 16:30 → 00:30).

type OllamaProvider 

type OllamaProvider struct {
	// contains filtered or unexported fields
}

func NewOllama 

func NewOllama(cfg Config) *OllamaProvider

func (*OllamaProvider) Complete 

func (p *OllamaProvider) Complete(ctx context.Context, req Request) (Response, error)

func (*OllamaProvider) IsLoaded 

func (p *OllamaProvider) IsLoaded(ctx context.Context, model string) bool

func (*OllamaProvider) Name 

func (p *OllamaProvider) Name() string

func (*OllamaProvider) Preload 

func (p *OllamaProvider) Preload(ctx context.Context, model string) error

func (*OllamaProvider) Stream 

func (p *OllamaProvider) Stream(ctx context.Context, req Request) (iter.Seq2[string, error], error)

Stream uses newline-delimited JSON, not SSE. Ollama's /api/chat with stream:true sends one JSON object per line, terminated by a `done:true` field.

func (*OllamaProvider) Unload 

func (p *OllamaProvider) Unload(ctx context.Context, model string) error

type OpenAIProvider 

type OpenAIProvider struct {
	// contains filtered or unexported fields
}

func NewOpenAI 

func NewOpenAI(cfg Config) *OpenAIProvider

func (*OpenAIProvider) Complete 

func (p *OpenAIProvider) Complete(ctx context.Context, req Request) (Response, error)

func (*OpenAIProvider) Name 

func (p *OpenAIProvider) Name() string

func (*OpenAIProvider) Stream 

func (p *OpenAIProvider) Stream(ctx context.Context, req Request) (iter.Seq2[string, error], error)

type Price 

type Price struct {
	PromptPer1k      float64 `json:"prompt_per_1k"            yaml:"prompt_per_1k"`
	PromptCacheHit1k float64 `json:"prompt_cache_hit_per_1k"  yaml:"prompt_cache_hit_per_1k"`
	CompPer1k        float64 `json:"comp_per_1k"              yaml:"comp_per_1k"`
}

Price is a per-1000-token rate for one billing tier. 

PromptPer1k      cache-MISS input rate (the "expensive" input path)
PromptCacheHit1k cache-HIT input rate. Providers that do not offer
                 prompt caching should leave this equal to
                 PromptPer1k, or zero when the field is unused.
CompPer1k        output rate

All values are in the currency declared by the enclosing PriceCard.

type PriceCard 

type PriceCard struct {
	RateCardID string        `json:"rate_card_id" yaml:"rate_card_id"`
	Currency   cost.Currency `json:"currency" yaml:"currency"`
	Peak       Price         `json:"peak"     yaml:"peak"`
	OffPeak    *Price        `json:"off_peak,omitempty" yaml:"off_peak,omitempty"`
}

PriceCard is the full rate card for one model in one currency. 

Peak     applied 24/7 unless OffPeak is set and the call happens
         inside the catalog's OffPeakWindow.
OffPeak  optional discounted rate card. Leave nil when the
         provider does not differentiate by time-of-day.
Currency the currency in which Peak and OffPeak are denominated.

type PriceCatalog 

type PriceCatalog struct {
	// contains filtered or unexported fields
}

PriceCatalog is the process-wide rate card registry. It is safe for concurrent readers and occasional writers (Register / reload).

func NewPriceCatalog 

func NewPriceCatalog() *PriceCatalog

NewPriceCatalog returns an empty-by-default catalog. Applications load their provider/model rate cards, currencies, and billing windows from configuration.

func (*PriceCatalog) Get 

func (c *PriceCatalog) Get(model string) (PriceCard, bool)

Get returns a model's card only when exactly one card is configured for it. Ambiguous model-only estimates fail closed; use GetRateCard with an explicit ID.

func (*PriceCatalog) GetRateCard added in v0.2.0 

func (c *PriceCatalog) GetRateCard(model, rateCardID string) (PriceCard, bool)

GetRateCard returns the exact card registered for the model and ID.

func (*PriceCatalog) IsOffPeak 

func (c *PriceCatalog) IsOffPeak(t time.Time) bool

IsOffPeak reports whether t falls inside the configured window. The window is UTC by contract.

func (*PriceCatalog) LoadFromFile 

func (c *PriceCatalog) LoadFromFile(filePath string) error

func (*PriceCatalog) LoadFromURL 

func (c *PriceCatalog) LoadFromURL(ctx context.Context, url string) error

func (*PriceCatalog) LoadFromYAML 

func (c *PriceCatalog) LoadFromYAML(data []byte) error

LoadFromYAML accepts an explicit models map with per-card or top-level currency. Every model entry must carry an explicit rate_card_id; cards without an explicit currency are rejected instead of assuming one.

func (*PriceCatalog) Models 

func (c *PriceCatalog) Models() []string

Models returns a sorted list of every model that has a rate card. Useful for introspection and startup banners.

func (*PriceCatalog) Register 

func (c *PriceCatalog) Register(model string, card PriceCard) error

Register installs or replaces a rate card for a model. Both the model and card identity are required; an incomplete card is never registered.

type Provider 

type Provider interface {
	Name() string
	Complete(ctx context.Context, req Request) (Response, error)
	Stream(ctx context.Context, req Request) (iter.Seq2[string, error], error)
}

func NewObservableProvider 

func NewObservableProvider(inner Provider, emit ProviderTrace) Provider

NewObservableProvider returns inner wrapped with emit. If either inner or emit is nil the function returns inner unchanged, so callers can chain it unconditionally.

func NewProvider 

func NewProvider(providerType string, cfg Config) (Provider, error)

NewProvider resolves and constructs a configured LLM Provider. Provider presets never select a model; applications supply it in Config or Request.

type ProviderChain 

type ProviderChain struct {
	// contains filtered or unexported fields
}

func NewProviderChain 

func NewProviderChain(primary Provider, fallbacks ...Provider) *ProviderChain

func (*ProviderChain) Complete 

func (c *ProviderChain) Complete(ctx context.Context, req Request) (Response, error)

func (*ProviderChain) Name 

func (c *ProviderChain) Name() string

func (*ProviderChain) Stream 

func (c *ProviderChain) Stream(ctx context.Context, req Request) (iter.Seq2[string, error], error)

type ProviderPreset 

type ProviderPreset struct {
	Name        string
	DefaultURL  string
	EnvKeyNames []string
	IsDeepSeek  bool
	IsInception bool
}

type ProviderTrace 

type ProviderTrace func(
	kind, provider, model string,
	in, out int,
	costMicro int64,
	dur time.Duration,
	err error,
)

ProviderTrace is the callback invoked by ObservableProvider around every completion attempt. 

kind       one of "start", "finish", "warn", "error"
provider   the provider's Name() (e.g. "deepseek")
model      the resolved model name; "auto" until known, then the
           server-reported name from the response
in, out    token counts (0 for "start" and "error")
costMicro  cost in the currency's micro units (0 when unknown)
dur        wall-clock time of the call (0 for "start")
err        non-nil for "error" and "warn"

type ReasoningEffort added in v0.2.0 

type ReasoningEffort string

ReasoningEffort is a provider/model-specific reasoning budget. Not all providers accept every value; unsupported values are passed to the API and reported using its normal typed provider error. 

const (
	ReasoningEffortNone    ReasoningEffort = "none"
	ReasoningEffortMinimal ReasoningEffort = "minimal"
	ReasoningEffortLow     ReasoningEffort = "low"
	ReasoningEffortMedium  ReasoningEffort = "medium"
	ReasoningEffortHigh    ReasoningEffort = "high"
	ReasoningEffortXHigh   ReasoningEffort = "xhigh"
	ReasoningEffortMax     ReasoningEffort = "max"
	ReasoningEffortInstant ReasoningEffort = "instant"
)

type Request 

type Request struct {
	Model string `json:"model,omitempty"`
	// The following fields are host accounting metadata. Provider adapters build
	// their own wire DTOs and must not send these fields to upstream APIs.
	CandidateID                 string          `json:"candidate_id,omitempty"`
	AttemptID                   string          `json:"attempt_id,omitempty"`
	RateCardID                  string          `json:"rate_card_id,omitempty"`
	ExpectedCurrency            *cost.Currency  `json:"expected_currency,omitempty"`
	EstimatedPromptTokens       int             `json:"estimated_prompt_tokens,omitempty"`
	EstimatedCachedPromptTokens int             `json:"estimated_cached_prompt_tokens,omitempty"`
	EstimatedCompletionTokens   int             `json:"estimated_completion_tokens,omitempty"`
	Messages                    []Message       `json:"messages"`
	Temperature                 float64         `json:"temperature,omitempty"`
	MaxTokens                   int             `json:"max_tokens,omitempty"`
	JSONMode                    bool            `json:"json_mode,omitempty"`
	OutputSchema                *JSONSchema     `json:"output_schema,omitempty"` // Optional schema constraint; currently applied by Anthropic Messages.
	ReasoningEffort             ReasoningEffort `json:"reasoning_effort,omitempty"`
	Tools                       []ToolSpec      `json:"tools,omitempty"`
	IncludeThinking             bool            `json:"include_thinking,omitempty"`
}

type Response 

type Response struct {
	Content            string        `json:"content"`
	ReasoningContent   string        `json:"reasoning_content,omitempty"`
	Model              string        `json:"model"`
	PromptTokens       int           `json:"prompt_tokens"`
	CachedPromptTokens int           `json:"cached_prompt_tokens,omitempty"`
	CompTokens         int           `json:"completion_tokens"`
	FinishReason       string        `json:"finish_reason,omitempty"`
	ToolCalls          []ToolCall    `json:"tool_calls,omitempty"`
	Duration           time.Duration `json:"duration,omitempty"`
}

func (Response) Cost 

func (r Response) Cost() (cost.Micro, cost.Currency, bool)

Cost returns both the amount and its currency.

func (Response) CostMicros 

func (r Response) CostMicros() cost.Micro

CostMicros returns the cost of this turn in micro-units of the currency reported by the rate card. The currency itself is available via Cost().

func (Response) Truncated 

func (r Response) Truncated() bool

Truncated reports whether the provider stopped because it hit the max_tokens ceiling. Callers should treat empty Content with Truncated()==true as "raise the limit", not "model failed".

type Role 

type Role string
const (
	RoleSystem    Role = "system"
	RoleUser      Role = "user"
	RoleAssistant Role = "assistant"
	RoleTool      Role = "tool"
)

type RouterSink 

type RouterSink struct {
	// contains filtered or unexported fields
}

RouterSink counts MoE router events. It is a plain observe.Sink so it can be attached with .AnyHook(observe.Hook(sink)) to any action set.

func NewRouterSink 

func NewRouterSink() *RouterSink

func (*RouterSink) Emit 

func (s *RouterSink) Emit(_ context.Context, ev observe.Event)

func (*RouterSink) Stats 

func (s *RouterSink) Stats() SinkStats

type SSEChunkDecoder 

type SSEChunkDecoder func(data []byte) (text string, done bool, err error)

SSEChunkDecoder turns the JSON payload of a single SSE `data:` frame into a text fragment. Providers differ only in this function.

Return values:

  • text: fragment to yield. Empty fragments are skipped by the skeleton.
  • done: true means the stream is complete and the reader stops.
  • err: decoding failed for this frame; the skeleton drops the frame and continues. A single malformed frame does not tear down a healthy stream.

type SinkStats 

type SinkStats struct {
	Executed uint64 `json:"executed"`
	Retries  uint64 `json:"retries"`
	Errors   uint64 `json:"errors"`
}

type StreamEvent 

type StreamEvent struct {
	Sequence   uint64          `json:"sequence"`
	Time       time.Time       `json:"time"`
	Kind       StreamEventKind `json:"kind"`
	RunID      string          `json:"run_id,omitempty"`
	SessionID  string          `json:"session_id,omitempty"`
	Turn       int             `json:"turn,omitempty"`
	Action     string          `json:"action,omitempty"`
	ToolCallID string          `json:"tool_call_id,omitempty"`
	ToolName   string          `json:"tool_name,omitempty"`
	Delta      string          `json:"delta,omitempty"`
	ToolResult string          `json:"tool_result,omitempty"`
	Response   *Response       `json:"response,omitempty"`
	Error      error           `json:"error,omitempty"`
}

StreamEvent represents an ephemeral lifecycle or streaming event.

type StreamEventKind 

type StreamEventKind uint8
const (
	EventRunStarted StreamEventKind = iota
	EventTurnStarted
	EventCompletionStarted
	EventTokenDelta
	EventCompletionFinished
	EventToolCallStarted
	EventToolCallFinished
	EventTurnFinished
	EventRunFinished
	EventError
)

func (StreamEventKind) String 

func (k StreamEventKind) String() string

type StreamSink 

type StreamSink interface {
	Emit(context.Context, StreamEvent) error
}

StreamSink receives zero-allocation streaming events across run boundaries.

type TieredModelRouter 

type TieredModelRouter struct {
	FastSmall Provider // Tier 1: Local / Groq (0$ or sub-cent)
	Standard  Provider // Tier 2: DeepSeek-V3 / GPT-4o-mini
	HeavyDeep Provider // Tier 3: Claude 3.5 Sonnet / o1 (High reasoning)
}

TieredModelRouter dynamically escalates queries to higher-tier models if smaller models fail.

func NewTieredModelRouter 

func NewTieredModelRouter(fast, standard, heavy Provider) *TieredModelRouter

func (*TieredModelRouter) CompleteAdaptive 

func (r *TieredModelRouter) CompleteAdaptive(ctx context.Context, req Request, minTier int) (Response, int, error)

CompleteAdaptive routes queries through increasing capability tiers.

type TokenEstimator 

type TokenEstimator interface {
	EstimateTokens(messages []Message) int
}

type ToolCall 

type ToolCall struct {
	ID       string       `json:"id"`
	Type     string       `json:"type"`
	Function FunctionCall `json:"function"`
}

type ToolSpec 

type ToolSpec struct {
	Type     string       `json:"type"`
	Function FunctionSpec `json:"function"`
}

type TypeSafeProvider added in v0.2.0 

type TypeSafeProvider struct {
	// contains filtered or unexported fields
}

TypeSafeProvider represents TypeSafe's native SystemOne decision endpoint. The documented API has no provider-enforced output-token cap, so Decide refuses dispatch rather than silently ignoring DecisionRequest.MaxOutputTokens.

func NewTypeSafe added in v0.2.0 

func NewTypeSafe(cfg Config) *TypeSafeProvider

func (*TypeSafeProvider) Decide added in v0.2.0 

func (p *TypeSafeProvider) Decide(_ context.Context, req DecisionRequest) (DecisionResponse, error)

func (*TypeSafeProvider) Name added in v0.2.0 

func (p *TypeSafeProvider) Name() string

type TypedReq 

type TypedReq struct {
	System string `json:"system,omitempty"`
	User   string `json:"user" validate:"required"`
}

TypedReq is the input contract for every NewTypedNode-produced action. A caller builds one of these, the action forwards it to the provider in JSON mode, and the model's reply is unmarshalled into Res.

The two fields are free-form strings so the node can be composed without knowing anything about the caller's domain. System carries the reviewer/architect/planner persona; User carries the payload rendered as text by an upstream projection.

type Usage 

type Usage struct {
	Model              string         `json:"model,omitempty"`
	RateCardID         string         `json:"rate_card_id,omitempty"`
	ExpectedCurrency   *cost.Currency `json:"expected_currency,omitempty"`
	PromptTokens       int            `json:"prompt_tokens,omitempty"`
	CachedPromptTokens int            `json:"cached_prompt_tokens,omitempty"`
	CompTokens         int            `json:"comp_tokens,omitempty"`
	FinishReason       string         `json:"finish_reason,omitempty"`
	Duration           time.Duration  `json:"duration,omitempty"`
	CostMicro          cost.Micro     `json:"cost_micros,omitempty"`
	Currency           *cost.Currency `json:"currency,omitempty"`
	CostKnown          bool           `json:"cost_known"`
}

Usage is the LLM accounting that travels with every agent result.

Any result that embeds Usage automatically satisfies the runner observer's token-cost interface, so the metrics table picks it up without per-result Tokens()/CostUSD() methods.

Usage is a copyable value object. Currency is a pointer only because unknown-rate-card results have no currency value.

func FromResponse 

func FromResponse(r Response) Usage

FromResponse extracts Usage from a completed Response.

func FromResponseForRateCard added in v0.2.0 

func FromResponseForRateCard(r Response, rateCardID string, expectedCurrency cost.Currency) Usage

FromResponseForRateCard extracts usage using the exact caller-selected card. A missing card or currency mismatch remains unknown; it never falls back to a model-only card lookup.

func (Usage) CostMicros 

func (u Usage) CostMicros() cost.Micro

CostMicros returns the cost in micro-units of the rate-card currency. Zero when the model has no rate card — that is intentional; the observer prints "-" rather than "0.000000" for unknown models.

func (Usage) CostUSD 

func (u Usage) CostUSD() int64

CostUSD is the compat method expected by the runner observer's tokenCostReporter interface. The value is micro-units of whatever currency the rate card declares; the name is historical. Callers that need the currency should call CalculateCost directly.

func (Usage) Tokens 

func (u Usage) Tokens() (prompt, completion int)

Tokens returns (input, output) token counts.

func (Usage) Truncated 

func (u Usage) Truncated() bool

Truncated reports whether the provider stopped because it hit the max-tokens ceiling.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.