Documentation
¶
Index ¶
- Constants
- type InternalHandleResult
- type InternalResponse
- type ResponseCacheMiddleware
- func (m *ResponseCacheMiddleware) Close() error
- func (m *ResponseCacheMiddleware) HandleInternalRequest(ctx context.Context, method, path string, body []byte, ...) (*InternalHandleResult, error)
- func (m *ResponseCacheMiddleware) HandleRequest(c *echo.Context, body []byte, next func() error) error
- func (m *ResponseCacheMiddleware) Ping(ctx context.Context) error
- func (m *ResponseCacheMiddleware) UsesRedis() bool
- type VecResult
- type VecStore
Constants ¶
const ( CacheTypeExact = "exact" CacheTypeSemantic = "semantic" CacheHeaderExact = "HIT (exact)" CacheHeaderSemantic = "HIT (semantic)" )
CacheTypeHeader values for X-Cache-Type.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type InternalHandleResult ¶
type InternalHandleResult struct {
StatusCode int
Headers http.Header
Body []byte
CacheType string
}
InternalHandleResult is the buffered result of running the cache middleware for a transport-free internal JSON request.
type InternalResponse ¶
InternalResponse is the buffered outcome of the transport-free LLM call a cache miss executes.
type ResponseCacheMiddleware ¶
type ResponseCacheMiddleware struct {
// contains filtered or unexported fields
}
ResponseCacheMiddleware wraps response cache logic. App and server only see this type.
func NewResponseCacheMiddleware ¶
func NewResponseCacheMiddleware( cfg config.ResponseCacheConfig, resolvedProviders map[string]config.RawProviderConfig, usageLogger usage.LoggerInterface, pricingResolver usage.PricingResolver, ) (*ResponseCacheMiddleware, error)
NewResponseCacheMiddleware creates middleware from config. If neither simple nor semantic cache is configured, returns a no-op middleware. resolvedProviders must be the credential/env-resolved map from providers.InitResult (same keys as live routing). Semantic embedder.provider names a key in this map.
func NewResponseCacheMiddlewareWithStore ¶
func NewResponseCacheMiddlewareWithStore(store cache.Store, ttl time.Duration) *ResponseCacheMiddleware
NewResponseCacheMiddlewareWithStore creates middleware with a custom store (for testing).
func (*ResponseCacheMiddleware) Close ¶
func (m *ResponseCacheMiddleware) Close() error
Close waits for any in-flight cache writes to complete, then releases cache resources.
func (*ResponseCacheMiddleware) HandleInternalRequest ¶
func (m *ResponseCacheMiddleware) HandleInternalRequest( ctx context.Context, method, path string, body []byte, next func(ctx context.Context) (*InternalResponse, error), ) (*InternalHandleResult, error)
HandleInternalRequest runs the cache for a transport-free internal JSON request. Request headers are derived from the originating request snapshot (allowlisted), and next executes the LLM call, returning a buffered response instead of writing to a socket.
func (*ResponseCacheMiddleware) HandleRequest ¶
func (m *ResponseCacheMiddleware) HandleRequest(c *echo.Context, body []byte, next func() error) error
HandleRequest runs the full dual-layer cache check (exact then semantic) for a translated inference request that has already been guardrail-patched. body is the final patched request bytes; next is the real LLM call. Streaming and non-streaming requests are cached independently. Streaming misses persist raw SSE bytes and replay them verbatim on cache hits.
func (*ResponseCacheMiddleware) Ping ¶
func (m *ResponseCacheMiddleware) Ping(ctx context.Context) error
Ping verifies connectivity to the Redis-backed exact cache. It returns nil when no Redis cache is configured, so callers should gate on UsesRedis first if they need to distinguish "not configured" from "reachable".
func (*ResponseCacheMiddleware) UsesRedis ¶
func (m *ResponseCacheMiddleware) UsesRedis() bool
UsesRedis reports whether a Redis-backed exact (simple) cache is configured. Only then is the cache a meaningful readiness component worth probing.
type VecStore ¶
type VecStore interface {
// Search returns up to limit results whose params_hash matches and whose
// similarity score to vec is above the caller's threshold. Expired entries
// encountered during search are deleted before returning a miss.
Search(ctx context.Context, vec []float32, paramsHash string, limit int) ([]VecResult, error)
// Insert stores vec along with its response bytes and TTL metadata.
// expires_at is recorded as unix-seconds; Search and DeleteExpired filter on it.
Insert(ctx context.Context, key string, vec []float32, response []byte, paramsHash string, ttl time.Duration) error
// DeleteExpired removes all entries whose expires_at is in the past.
// Called periodically by a background goroutine started in NewVecStore.
DeleteExpired(ctx context.Context) error
Close() error
}
VecStore is the generic interface for all vector database backends. Each backend must implement similarity search, insertion with TTL metadata, and bulk expired-entry deletion for background cleanup.
func NewVecStore ¶
func NewVecStore(cfg config.VectorStoreConfig) (VecStore, error)
NewVecStore creates a VecStore for the backend selected by cfg.Type. Type must be non-empty and one of: qdrant, pgvector, pinecone, weaviate.
Source Files