Documentation
¶
Overview ¶
Package provider is one client for every model runtime.
Ollama, LM Studio, Jan and Msty all speak OpenAI-compatible /v1, so the only thing that differs is the port and which models happen to be loaded. Cloud BYOK is the same surface with a base URL and a key.
Index ¶
- Constants
- Variables
- func Off() bool
- type Discovered
- type LocalEndpoint
- type Msg
- type Provider
- func (p *Provider) Chat(model, system, user string, schema map[string]any) (string, error)
- func (p *Provider) ChatStream(model string, messages []Msg, onToken func(string)) (string, error)
- func (p *Provider) Embed(model string, inputs []string) ([][]float32, error)
- func (p *Provider) Interactive(timeout time.Duration) *Provider
- func (p *Provider) Stalls() int
Constants ¶
const StallCooloff = time.Minute
StallCooloff is how long an Interactive provider skips embeddings after one timed out.
Variables ¶
var LocalEndpoints = []LocalEndpoint{
{"Ollama", "http://localhost:11434/v1"},
{"LM Studio", "http://localhost:1234/v1"},
{"Jan", "http://localhost:1337/v1"},
{"Msty", "http://localhost:10000/v1"},
}
Probed in this order; first match wins when nothing is configured.
Functions ¶
func Off ¶ added in v0.5.2
func Off() bool
Off reports LOGOS_RUNTIME=off: no runtime, whatever answers on localhost. The ports are fixed, so without it a machine with Ollama running could not show — to a test, a first-run script, or a user deciding whether they need a model — what logos does with none.
Types ¶
type Discovered ¶
func Discover ¶
func Discover() []Discovered
Discover probes the well-known ports and reports what is actually running. This is what lets the app say "found LM Studio running Qwen3" on first launch instead of presenting an empty endpoint configuration box.
func Resolve ¶ added in v0.4.4
func Resolve() []Discovered
Resolve is Discover, unless LOGOS_RUNTIME names a runtime, in which case that runtime is the only candidate. It is what every long-lived caller resolves through. `logos index` honoured LOGOS_RUNTIME and `mcp serve` did not, so the server embedded queries on whatever answered on localhost — or, with the configured runtime on another host, had no embeddings at all — while the index had been built somewhere else.
A configured runtime that does not answer is reported as none rather than falling back to discovery: the user named where the model runs, and quietly using a different runtime is how vectors end up from two models.
type LocalEndpoint ¶
LocalEndpoint is a runtime we know how to find without configuration.
type Provider ¶
type Provider struct {
Name string
BaseURL string
APIKey string
// Think controls how much a reasoning model reasons before answering:
// "off", "low", "medium", "high" (empty = "low"). It matters because the
// OpenAI-compatible /v1 endpoint gives a reasoning model no way to bound its
// thinking, so it spends the whole token budget thinking and returns an empty
// answer. When set (Ollama only), chat routes through the native /api/chat
// endpoint, which honours it. "default" forces the plain /v1 path.
Think string
// contains filtered or unexported fields
}
func Configured ¶ added in v0.4.4
func Configured() *Provider
Configured is the runtime LOGOS_RUNTIME names (LOGOS_RUNTIME_KEY for a bearer token), or nil when it is unset or off. Discovery only probes localhost ports, so this is the only way to reach a runtime on another host or a non-standard port.
func (*Provider) Chat ¶
Chat runs a completion. A non-nil schema enables constrained decoding: small local models are unreliable at tool-calling but near-perfect when JSON is enforced at the sampler, which is why every extraction path goes through here rather than through tool definitions.
func (*Provider) ChatStream ¶
ChatStream runs a multi-turn completion and streams tokens to onToken as they arrive. This is what makes the assistant feel alive: a 40-second answer that appears word by word reads as thinking, while the same answer delivered in one silent lump reads as broken.
Returns the full assembled text so callers can persist the turn.
func (*Provider) Interactive ¶ added in v0.4.4
Interactive is a copy of p for embeddings someone is waiting on. The 300 s client is sized for generation, and embeddings shared it: a runtime that listed its models and then never answered — loading the embedding model behind a large one, or wedged after sleep — held an MCP tool call for minutes, three times over in resume, and the host showed a timeout with nothing naming the model as the cause. A refused connection already failed at once; this makes a silent runtime fail the same way after timeout, and then treat it as down for StallCooloff so the calls after it don't each pay again.
Source Files