Documentation
¶
Overview ¶
Package cachepolicy describes what prompt caching a provider actually supports, so callers are not misled about whether caching happened.
Providers differ fundamentally, and an abstraction that hides that does more harm than good:
- Anthropic caches on explicit cache_control breakpoints the caller places.
- OpenAI and Azure OpenAI cache automatically, with no control surface, but do report how many prompt tokens were served from cache.
- Gemini has both implicit caching and a separate explicit cached-content API with its own lifecycle.
- Several providers do nothing at all.
A single CacheConfig applied uniformly would therefore be a no-op on most providers while looking configured. This package makes the difference inspectable instead: ask what a provider supports before assuming a setting took effect.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func CheckConfig ¶
CheckConfig reports whether an explicit caching configuration will do anything on a provider.
Call it at startup rather than discovering from a flat bill that a setting was inert. Returns nil when the configuration is honored or when no caching was requested.
Types ¶
type Capability ¶
type Capability struct {
// Provider is the provider name, matching interfaces.LLM.Name().
Provider string
// Control is how much say the caller has.
Control Support
// ReportsUsage is whether cache hits appear in TokenUsage.
//
// This is tracked separately from Control on purpose: OpenAI offers no
// control at all yet reports cache reads, which is exactly the combination
// a single "supports caching" boolean would misrepresent.
ReportsUsage bool
// Notes explains anything a caller would otherwise get wrong.
Notes string
}
Capability describes one provider's caching behaviour.
func For ¶
func For(provider string) Capability
For returns the caching capability of a provider.
An unknown provider reports no support, which is the safe answer: assuming a provider caches when it does not leads a caller to expect savings that never arrive.
func (Capability) Honors ¶
func (c Capability) Honors() bool
Honors reports whether an explicit CacheConfig has any effect here.
Source Files