Documentation
¶
Overview ¶
Package runtimestate reconciles the declared state of LLM backends (from dbInstance) with their actual observed state. It keeps runtime observation read-only and is intended to be executed repeatedly within background tasks managed externally.
The central type is State (constructed via New), which other packages — notably llmrepo — consult through ProviderFromRuntimeState to enumerate currently-available modelrepo.Provider instances. Init{Embeder,PromptExec, ChatExec} bootstrap default model configurations from a Config; provider subpackages register their catalogs via blank imports.
Index ¶
- Constants
- func ClearModelCache(ctx context.Context, kv libkvstore.KVManager) (int, error)
- func EnsureModels(ctx context.Context, dbInstance libdb.DBManager, tenantID string, ...) error
- func InitChatExec(ctx context.Context, config *Config, dbInstance libdb.DBManager, ...) error
- func InitEmbeder(ctx context.Context, config *Config, dbInstance libdb.DBManager, ...) error
- func InitPromptExec(ctx context.Context, config *Config, dbInstance libdb.DBManager, ...) error
- func InvalidateModelCache(ctx context.Context, kv libkvstore.KVManager, backendID string) error
- type Config
- type ExtraModelSpec
- type Option
- type ProviderConfig
- type ProviderFromRuntimeState
- type State
Constants ¶
const ( ProviderKeyPrefix = "cloud-provider:" OllamaKey = ProviderKeyPrefix + "ollama" OpenaiKey = ProviderKeyPrefix + "openai" OpenRouterKey = ProviderKeyPrefix + "openrouter" AnthropicKey = ProviderKeyPrefix + "anthropic" MistralKey = ProviderKeyPrefix + "mistral" BedrockKey = ProviderKeyPrefix + "bedrock" GeminiKey = ProviderKeyPrefix + "gemini" VertexGoogleKey = ProviderKeyPrefix + "vertex-google" )
const ProviderCacheDuration = 1 * time.Hour
ProviderCacheDuration defines how long the state of models from an external provider (like OpenAI or Gemini) is cached to avoid frequent API calls.
Variables ¶
This section is empty.
Functions ¶
func ClearModelCache ¶
ClearModelCache removes every cached observed-model list (all "prov:*" keys) and returns how many were cleared. No-op (0) when kv is nil.
func EnsureModels ¶
func EnsureModels(ctx context.Context, dbInstance libdb.DBManager, tenantID string, specs []ExtraModelSpec) error
EnsureModels ensures each given model exists in ollama_models with the specified context length and capabilities. It is intended for contenox-cli so that extra models (e.g. qwen2.5:7b for the vibes chain) are declared and get correct context/capabilities during backend sync. Call after InitEmbeder/InitPromptExec/InitChatExec and before RunBackendCycle.
func InitChatExec ¶
func InitChatExec(ctx context.Context, config *Config, dbInstance libdb.DBManager, runtime *State, contextLen int) error
InitChatExec initializes the chat group and its designated model.
func InitEmbeder ¶
func InitEmbeder(ctx context.Context, config *Config, dbInstance libdb.DBManager, contextLen int, runtime *State) error
InitEmbeder initializes the embedding group and its designated model.
func InitPromptExec ¶
func InitPromptExec(ctx context.Context, config *Config, dbInstance libdb.DBManager, runtime *State, contextLen int) error
InitPromptExec initializes the tasks group and its designated model.
func InvalidateModelCache ¶
InvalidateModelCache removes the cached observed-model list for one backend, forcing the next backend cycle to refetch from the provider. Safe when no entry exists; no-op when kv is nil.
Types ¶
type Config ¶
type Config struct {
DatabaseURL string `json:"database_url"`
EmbedModel string `json:"embed_model"`
TaskModel string `json:"task_model"`
ChatModel string `json:"chat_model"`
TenantID string `json:"tenant_id"`
}
Config holds the configuration for the runtime state initializer.
type ExtraModelSpec ¶
type ExtraModelSpec struct {
Name string
ContextLength int
CanChat bool
CanPrompt bool
CanEmbed bool
}
ExtraModelSpec describes an extra model to ensure exists in ollama_models (e.g. for contenox-cli extra_models config).
type Option ¶
type Option func(*State)
func WithAutoDiscoverModels ¶
func WithAutoDiscoverModels() Option
WithAutoDiscoverModels exposes all models returned by live backends without requiring manual declaration via 'model add'. Capability inference remains name-based for providers (e.g. OpenAI) whose APIs do not return capability metadata.
func WithGroups ¶
func WithGroups() Option
func WithKVStore ¶
func WithKVStore(kv libkvstore.KVManager) Option
WithKVStore injects a persistent KV store for provider model-list caching. For the CLI use libkvstore.NewSQLiteManager; for the runtime API use libkvstore.NewManager (Valkey). When not provided the cache falls back to an in-memory sync.Map.
func WithSkipDeleteUndeclaredModels ¶
func WithSkipDeleteUndeclaredModels() Option
WithSkipDeleteUndeclaredModels is kept as a no-op compatibility option. OSS runtime reconciliation is observation-only and no longer deletes backend models.
type ProviderConfig ¶
func (ProviderConfig) MarshalJSON ¶
func (pc ProviderConfig) MarshalJSON() ([]byte, error)
type ProviderFromRuntimeState ¶
type ProviderFromRuntimeState func(ctx context.Context, backendTypes ...string) ([]modelrepo.Provider, error)
ProviderFromRuntimeState retrieves available model providers
func LocalProviderAdapter ¶
func LocalProviderAdapter(ctx context.Context, tracker libtracker.ActivityTracker, runtime map[string]statetype.BackendRuntimeState) ProviderFromRuntimeState
LocalProviderAdapter creates providers for self-hosted backends (Ollama, vLLM)
type State ¶
type State struct {
// contains filtered or unexported fields
}
State manages the overall runtime status of multiple LLM backends. It orchestrates synchronization between the desired configuration and the actual observed state of the backends.
func New ¶
func New(ctx context.Context, dbInstance libdb.DBManager, psInstance libbus.Messenger, options ...Option) (*State, error)
New creates and initializes a new State manager. It requires a database manager (dbInstance) to load the desired configurations and a messenger instance (psInstance) for event handling and progress updates. Options allow enabling experimental features like group-based reconciliation. Returns an initialized State ready for use.
func (*State) Get ¶
Get returns a copy of the current observed state for all backends. This provides a safe snapshot for reading state without risking modification of the internal structures.
func (*State) RunBackendCycle ¶
RunBackendCycle performs a single reconciliation check for all configured LLM backends. It compares the desired state (from configuration) with the observed state (by communicating with the backends) and refreshes the runtime snapshot. This method should be called periodically in a background process. DESIGN NOTE: This method executes one complete reconciliation cycle and then returns. It does not manage its own background execution (e.g., via internal goroutines or timers). This deliberate design choice delegates execution management (scheduling, concurrency control, lifecycle via context, error handling, circuit breaking, etc.) entirely to the caller.
Consequently, this method should be called periodically by an external process responsible for its scheduling and lifecycle. When the group feature is enabled via Withgroups option, it uses group-aware reconciliation.
Source Files