Documentation
¶
Overview ¶
Package modeldata keeps the files the LLM gateway engine prices requests and looks up model capabilities from: a pricing datasheet and a model-parameters file.
By default the engine downloads both from its vendor's website. Obot instead keeps them in the gateway's data dir and points the engine at them with file:// URLs, so the engine never fetches them itself and always has something to boot with: with an empty store and nothing to load, the engine exits the process.
The files are converted from LiteLLM's model_prices_and_context_window.json (MIT; see data/LICENSE.litellm). Obot's images and release binaries embed a snapshot downloaded at build time (see Bundle), and Obot refreshes the files from a URL that serves the same conversion, Obot's upgrade server by default. The engine re-reads them on its own schedule.
Index ¶
- Constants
- func Bundle(ctx context.Context, baseURL, dir string) error
- func Download(ctx context.Context, client *http.Client, dataDir, baseURL string) error
- func FileURL(path string) string
- func Paths(dataDir string) (pricing, params string)
- func Prepare(ctx context.Context, dataDir, refreshURL string) error
- func Refresh(ctx context.Context, dataDir, baseURL string, interval time.Duration)
- func RefreshURL(raw string) (string, error)
Constants ¶
const ( // RefreshInterval is how often Obot downloads the files from the refresh URL. RefreshInterval = 6 * time.Hour )
Variables ¶
This section is empty.
Functions ¶
func Bundle ¶
Bundle downloads the model data baseURL serves into dir, compressed, for a build to embed.
func Download ¶
Download replaces the gateway's model data files with the ones baseURL serves, if both pass validation.
func Prepare ¶
Prepare makes sure the gateway's data dir has model pricing and capability files for the engine to read.
- With a bundled snapshot, the snapshot is written unless the files there are newer, as they are once a refresh has downloaded them.
- Without one, as in a build from source, files already there are kept. Otherwise Prepare tries to download them from refreshURL, and failing that writes empty files: the gateway starts with no model prices until a refresh succeeds.
func Refresh ¶
Refresh downloads the model pricing and capability files from baseURL every interval until ctx ends. A download that fails or doesn't pass validation leaves the current files in place.
func RefreshURL ¶
RefreshURL validates the configured model data URL. Empty or "off" turns refreshing off.
Types ¶
This section is empty.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package modeldatatest provides model data for tests: a few real entries, converted the way Obot's model data is, so tests that need model prices don't depend on the data a build downloads.
|
Package modeldatatest provides model data for tests: a few real entries, converted the way Obot's model data is, so tests that need model prices don't depend on the data a build downloads. |