Documentation
¶
Overview ¶
Package live caches the response of provider /v1/models calls per (provider, keyID, unfiltered). Filtered entries are pre-gated by the provider's ListModelsPipeline against the key's allowed/blacklisted/aliases; callers reading filtered entries MUST NOT reapply that gate elsewhere or alias-backfill rows will be dropped.
The store is passive: it never calls the network. Callers decide when to fetch and push results in via Upsert. Today those are the HTTP server's bootstrap seed, its key add/update handlers, the on-demand refresh endpoints, and BifrostHTTPServer.startLiveModelRefresher, which re-fetches every provider on the configured live_models_sync_interval unless that interval is 0, which disables the refresher.
Entries are per process and are never persisted, so every node refreshes its own copy rather than electing one refresher.
Index ¶
- type Entry
- type Key
- type Store
- func (s *Store) Generation(provider schemas.ModelProvider) uint64
- func (s *Store) Invalidate(provider schemas.ModelProvider, keyID string)
- func (s *Store) InvalidateProvider(provider schemas.ModelProvider)
- func (s *Store) ModelsForProvider(provider schemas.ModelProvider) []string
- func (s *Store) RetainKeys(provider schemas.ModelProvider, keep map[string]struct{})
- func (s *Store) Snapshot() map[Key]Entry
- func (s *Store) UnfilteredModelsForProvider(provider schemas.ModelProvider) []string
- func (s *Store) Upsert(provider schemas.ModelProvider, keyID string, unfiltered bool, models []string)
- func (s *Store) UpsertIfCurrent(provider schemas.ModelProvider, keyID string, unfiltered bool, models []string, ...) bool
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Key ¶
type Key struct {
Provider schemas.ModelProvider
KeyID string
Unfiltered bool
}
Key identifies one cached response. KeyID is "" for keyless providers (Vertex workload identity, Bedrock IAM, etc).
type Store ¶
type Store struct {
// contains filtered or unexported fields
}
func (*Store) Generation ¶ added in v1.5.5
func (s *Store) Generation(provider schemas.ModelProvider) uint64
Generation returns the provider's current invalidation counter. Read it before starting a fetch and hand it back to UpsertIfCurrent.
func (*Store) Invalidate ¶
func (s *Store) Invalidate(provider schemas.ModelProvider, keyID string)
Invalidate drops both filtered and unfiltered entries for one key. Called when the key's credential value changes (cached models were computed against the old credential) or when the key is deleted.
func (*Store) InvalidateProvider ¶
func (s *Store) InvalidateProvider(provider schemas.ModelProvider)
InvalidateProvider drops every entry for the provider across all keys and modes. Called on provider delete.
func (*Store) ModelsForProvider ¶
func (s *Store) ModelsForProvider(provider schemas.ModelProvider) []string
ModelsForProvider returns the union of filtered entries for the provider, sorted. Filtered entries are pre-gated so this is the effective allowed set across the provider's keys.
func (*Store) RetainKeys ¶ added in v1.5.5
func (s *Store) RetainKeys(provider schemas.ModelProvider, keep map[string]struct{})
RetainKeys drops the provider's entries whose KeyID is absent from keep, in both filtered and unfiltered modes, and leaves the rest untouched. It is the narrow counterpart to InvalidateProvider: callers about to re-fetch use it to prune keys that were removed or disabled while letting the surviving keys' last-known-good entries stand until a successful Upsert replaces them. A key named in keep that has no entry is a no-op, so callers can pass their whole configured key set without first checking what is cached.
keep is read-only and is not retained by the store. A nil or empty keep is equivalent to InvalidateProvider.
Bumps the provider's generation like the other invalidations, which costs the caller nothing: every caller re-fetches straight afterwards and so captures the new generation. A concurrent background pass loses that provider's results for one cycle, which is the correct outcome — the caller's own re-fetch replaces them, and the pass was working from the pre-prune key set.
func (*Store) Snapshot ¶
Snapshot returns a defensive copy of every entry for diagnostics. Slices are copied; the returned map is independent of store state.
func (*Store) UnfilteredModelsForProvider ¶
func (s *Store) UnfilteredModelsForProvider(provider schemas.ModelProvider) []string
UnfilteredModelsForProvider returns the union of unfiltered entries — the raw provider catalog with no key-level gating applied.
func (*Store) Upsert ¶
func (s *Store) Upsert(provider schemas.ModelProvider, keyID string, unfiltered bool, models []string)
Upsert stores a successful fetch unconditionally. Use it for writes with no in-flight window to lose a race in (seeding, tests); anything that fetches from an upstream first should go through Generation + UpsertIfCurrent.
func (*Store) UpsertIfCurrent ¶ added in v1.5.5
func (s *Store) UpsertIfCurrent(provider schemas.ModelProvider, keyID string, unfiltered bool, models []string, gen uint64) bool
UpsertIfCurrent stores a fetch only if nothing invalidated the provider since gen was read, and reports whether it wrote.
This is what keeps a deleted or disabled key's models from being resurrected by a fetch that outlived it. The background refresher works from a key snapshot taken at the top of a pass; by the time list-models answers, the key may be gone. Its Invalidate already ran, and no later pass re-fetches or prunes a key that is no longer configured, so an unguarded commit would advertise that key's models until the process restarted.
Source Files