Documentation
¶
Overview ¶
Package runtime provides Runtime, the one value that owns an embedding product's provider state: the provider registry, the provider cache, the credential refresh coordinators, the catalog cache and provider health.
The package keeps no package state and reads no environment. A product supplies everything through Options, including its own settings type. The Runtime never interprets settings; it notices when their generation changes and hands the snapshot to the product's factories.
Import it under another name, such as coreruntime, in a file that also uses the standard library's runtime package.
Index ¶
- Constants
- type MemorySettings
- type Options
- type Policy
- type PolicyFactory
- type ProviderFactory
- type RefreshFactory
- type RegistryInstance
- type Runtime
- func (r *Runtime[S]) CachedCatalog(ctx context.Context, caller core.Caller, instance string) catalog.Read
- func (r *Runtime[S]) CachedModel(ctx context.Context, instance, credentialKey, model string) (core.ModelInfo, bool)
- func (r *Runtime[S]) CatalogService() *catalog.Service
- func (r *Runtime[S]) Health(instance string) core.ProviderHealthEvidence
- func (r *Runtime[S]) Invoke(ctx context.Context, caller core.Caller, instance string, request core.Request) (core.Response, error)
- func (r *Runtime[S]) ListModels(ctx context.Context, caller core.Caller, instance string) (core.CatalogRecord, error)
- func (r *Runtime[S]) ReadCatalog(ctx context.Context, caller core.Caller, instance string) catalog.Read
- func (r *Runtime[S]) Registry() *providers.Registry
- func (r *Runtime[S]) Stream(ctx context.Context, caller core.Caller, instance string, request core.Request) (core.StreamIter, error)
- func (r *Runtime[S]) Transparent(ctx context.Context, caller core.Caller, instance string, request core.Request) (core.Response, error)
- func (r *Runtime[S]) TransportMode(ctx context.Context, caller core.Caller, instance, model string, ...) string
- type SettingsSource
Constants ¶
const DefaultCatalogTTL = 15 * time.Minute
DefaultCatalogTTL is how long a stored catalog is served before the Runtime rediscovers it, when Options.CatalogTTL is zero.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type MemorySettings ¶
type MemorySettings[S any] struct { // contains filtered or unexported fields }
MemorySettings is the in-memory reference SettingsSource.
func NewMemorySettings ¶
func NewMemorySettings[S any](settings S) *MemorySettings[S]
NewMemorySettings returns settings at generation 1.
func (*MemorySettings[S]) Snapshot ¶
func (m *MemorySettings[S]) Snapshot() (S, uint64)
Snapshot implements SettingsSource.
func (*MemorySettings[S]) Update ¶
func (m *MemorySettings[S]) Update(settings S) uint64
Update replaces the settings and returns the new generation.
type Options ¶
type Options[S any] struct { Settings SettingsSource[S] Providers ProviderFactory[S] // Credentials resolves and stores credentials. Nil sends every request // without one. Credentials core.CredentialStore // Refresh supplies each instance's OAuth refresh. Nil means credentials // never refresh. Refresh RefreshFactory[S] // Policy supplies each instance's retry and circuit policy, which the // Runtime applies to the provider it binds. Nil applies none. Policy PolicyFactory[S] // Catalogs stores discovered catalogs. Nil keeps them in this Runtime's // memory only. Catalogs core.CatalogStore // Evidence receives account evidence. Nil discards it. Evidence core.EvidenceSink // Registry is the product's effective provider registry. Nil uses // providers.DefaultRegistry. Registry *providers.Registry // CatalogTTL bounds how long a stored catalog is served. Zero uses // DefaultCatalogTTL. CatalogTTL time.Duration // CatalogService keeps catalogs instead, with its own store and TTLs, // and may keep stale rows. Catalogs and CatalogTTL are then unused. CatalogService *catalog.Service // Now returns the current time. Nil uses time.Now. Now func() time.Time }
Options configures a Runtime. Settings and Providers are required.
type Policy ¶ added in v0.14.0
type Policy struct {
// Retry repeats an operation whose failure permits a repeat. Below two
// attempts nothing repeats.
Retry execution.Retry
// Circuit is the instance's circuit policy. The Runtime keeps every
// instance's circuit in one execution.HealthTracker on its clock, keyed
// by instance, so a circuit is shared by every caller and outlives
// settings changes. A FailureThreshold of zero or less leaves the
// instance without a circuit: nothing is recorded or refused, Retry-After
// cooldowns included.
Circuit execution.HealthPolicy
// Classify and Repeatable are execution.Policy's.
Classify func(error) core.ProviderErrorClassification
Repeatable func(core.Request) bool
}
Policy is how the Runtime guards the provider of one instance with execution.Resilient: how often an operation is tried, and the instance's circuit. The zero Policy guards nothing, and the provider is used as built.
The gateway's per-provider policy maps onto it field for field: Retry is retry_max_attempts with execution.ExponentialBackoff of the backoff settings; Circuit is circuit_failure_threshold and circuit_cooldown_seconds with IgnoreRetryAfter; and Classify is execution.TransientClassification.
type PolicyFactory ¶ added in v0.14.0
PolicyFactory returns the resilience policy of one instance from a settings snapshot. It constructs and must not block: the Runtime calls it under its lock when it binds the instance's provider, once per instance and settings generation.
type ProviderFactory ¶
ProviderFactory builds the provider of one instance from a settings snapshot. It constructs and must not block: the Runtime calls it under its lock, once per instance and settings generation.
func RegistryProviders ¶ added in v1.8.0
func RegistryProviders[S any](registry *providers.Registry, describe func(settings S, instance string) (RegistryInstance, error)) ProviderFactory[S]
RegistryProviders returns a ProviderFactory that builds each instance from its registry entry: describe tells the instance's entry and settings, registry resolves the entry, nil meaning providers.DefaultRegistry, and providers.Build makes its provider. A product that builds its providers this way serves a runtime type core adds by upgrading, with no change to describe.
An entry the registry does not hold fails as providers.Build fails on an entry it cannot make a provider for: a configuration error, which permits failover, whose cause is a *providers.UnsupportedEntryError. An error describe returns is returned as it is.
type RefreshFactory ¶
type RefreshFactory[S any] func(settings S, instance string) tokenstore.RefreshFunc
RefreshFactory returns how one instance refreshes an OAuth credential, or nil when its credentials never refresh.
type RegistryInstance ¶ added in v1.8.0
type RegistryInstance struct {
// Entry is the id, or an alias, of the instance's registry entry.
Entry string
// Instance is how the instance is reached; see providers.Instance.
Instance providers.Instance
// Translate puts a translation.Adapter with Policy in front of the
// provider, so it also serves Chat Completions, Messages and Responses
// over whichever of them it speaks.
Translate bool
Policy core.LossPolicy
}
RegistryInstance is one provider instance as a product's settings describe it: the registry entry it serves, how it is reached, and whether a translation.Adapter serves the chat surfaces its provider does not speak.
type Runtime ¶
type Runtime[S any] struct { // contains filtered or unexported fields }
Runtime performs provider operations for callers. It is safe for concurrent use; keep one per process.
func (*Runtime[S]) CachedCatalog ¶ added in v0.14.0
func (r *Runtime[S]) CachedCatalog(ctx context.Context, caller core.Caller, instance string) catalog.Read
CachedCatalog returns the stored catalog ListModels would serve caller, fresh or not. It discovers nothing and refreshes no credential, so a read path stays safe while an upstream is slow or down.
func (*Runtime[S]) CachedModel ¶ added in v0.14.0
func (r *Runtime[S]) CachedModel(ctx context.Context, instance, credentialKey, model string) (core.ModelInfo, bool)
CachedModel returns the row of model in the stored catalog of instance that credentialKey discovered, without discovering it: the lookup a vertical's catalog hook, such as providers.ZenConfig.Models, and transport planning need.
func (*Runtime[S]) CatalogService ¶ added in v0.14.0
CatalogService returns the service that keeps this Runtime's catalogs, for cached reads and invalidation by key.
func (*Runtime[S]) Health ¶
func (r *Runtime[S]) Health(instance string) core.ProviderHealthEvidence
Health returns the last observed health of an instance.
func (*Runtime[S]) Invoke ¶
func (r *Runtime[S]) Invoke(ctx context.Context, caller core.Caller, instance string, request core.Request) (core.Response, error)
Invoke performs one non-streaming operation for caller on instance. An OAuth credential the upstream rejects is refreshed once and the request replayed.
func (*Runtime[S]) ListModels ¶
func (r *Runtime[S]) ListModels(ctx context.Context, caller core.Caller, instance string) (core.CatalogRecord, error)
ListModels returns the catalog of instance for caller. A stored catalog is served while it is fresh: younger than the catalog TTL, observed after the last settings change, and not a failure. Another process's discovery counts, because catalogs are shared through the CatalogStore and keyed by the credential that discovered them. Discoveries of one catalog are serialized within this Runtime, and fenced against invalidation (see ReadCatalog).
func (*Runtime[S]) ReadCatalog ¶ added in v0.14.0
func (r *Runtime[S]) ReadCatalog(ctx context.Context, caller core.Caller, instance string) catalog.Read
ReadCatalog is ListModels with the read's diagnostics. The catalog keeps its rows unless a discovery succeeds: a discovery that fails leaves them, and one that an invalidation fenced stores nothing (see catalog.Service.Invalidate). With a CatalogService that keeps stale rows, Record holds them beside the error of a failed discovery.
func (*Runtime[S]) Stream ¶
func (r *Runtime[S]) Stream(ctx context.Context, caller core.Caller, instance string, request core.Request) (core.StreamIter, error)
Stream performs one streaming operation for caller on instance. A rejected OAuth credential is refreshed and the stream reopened only while no frame has been delivered, that is when opening the stream fails.
func (*Runtime[S]) Transparent ¶ added in v0.14.0
func (r *Runtime[S]) Transparent(ctx context.Context, caller core.Caller, instance string, request core.Request) (core.Response, error)
Transparent performs one non-streaming request on instance exactly as the caller sent it. The body reaches the provider as is: no translation, no retry and no failover, only the credential refresh Invoke does.
It serves only what the caller's stored catalog confirms, as the gateway's transparent mode does: the catalog lists the model and the request's surface, the provider preserves that surface's wire for the model (see core.PreservesWire), and the row's capabilities are trusted and fresh (see core.TransportEvidence). It never discovers a catalog. Anything else, and then a body that asks to stream, fails with a *core.TransportRejectError before anything is sent.
func (*Runtime[S]) TransportMode ¶ added in v0.14.0
func (r *Runtime[S]) TransportMode(ctx context.Context, caller core.Caller, instance, model string, surface core.ModelSurface) string
TransportMode labels the transport of a response instance served caller for model on surface, as core.ResponseTransportMode does over the caller's stored catalog. It discovers nothing.
type SettingsSource ¶
SettingsSource supplies a product's current settings and a generation that changes whenever they do. After a generation change the Runtime rebuilds providers and refresh coordinators on their next use, and rediscovers catalogs observed before the change, so hot-reloaded settings take effect without a restart.
Source Files