cache

Cache storage

Application composition uses process memory for response, model, and extraction caches, independent of the durable storage backend. The cache manager accepts a cache-owned interface. It cannot accept the application's durable KV handle directly. An injected store requires the explicit distributed strategy. The manager owns that store after successful construction.

The default Ristretto capacities are 256 MiB for responses, 16 MiB for models, and 16 MiB for extraction records. These capacities bound cache cost, not total process memory. Extraction caching retains its independent lifecycle when the operator disables response caching.

CSP12.1 must qualify shared-service configuration, bounded fills, and stream accumulation limits.

Shared response bytes use the dedicated cache service directly. No local refill can extend their server expiry. The package has no durable KV cache adapter, layered cache, or invalidation pub/sub implementation. Real-service tests cover shared expiry and isolation. Local tests cover expiry and clearing. Historical evidence retains the removed adapters' original defects and repairs.

No production qualification or latency improvement follows from this interface change alone.

Optional response fills

Response and model fills share two lifecycle-owned workers. The queue limits queued and active work to 1,024 entries and 4 MiB of charged bytes. The byte charge includes copied keys, copied payloads, and 96 bytes per job. It does not measure total process or transport memory. Admission drops optional work when capacity or the admission lock is unavailable. SetResponse does not confirm persistence or a subsequent cache hit.

Writes have a 500 ms context deadline. Cache-only adapters must honor cancellation for blocking work. Shutdown cancels fills and joins workers before closing the stores. The admin info response reports limits, retained work, completed fills, failures, and drops under response_cache. It exposes no cache keys or payloads.

Injected response-cache reads have a 2 ms deadline. Local response reads use no additional timer. Current authorization and catalog checks still govern cached delivery. Model invalidation advances a process-local epoch. Queued writes from earlier epochs cannot become visible after invalidation. Model serialization still runs on the caller before queue admission.

The encoder rejects strings and byte slices above 512 KiB and containers above 65,536 entries before encoding their contents. Encoded output cannot exceed the fill budget after key and job charges. Model fills transfer their encoded buffer and charge its capacity. Model, provider, and endpoint reads decode directly into the requested response type.

Extraction fills use an independent queue with the same bounds, even when the operator disables response caching. Together, both queues permit at most 2,048 queued or active entries, 8 MiB of charged data, and four workers. The admin extraction_cache field reports extraction pressure. response_cache reports the shared response and model queue.

Extraction serialization remains synchronous for eligible records. The document cache rejects inputs above 512 KiB before serialization. That input bound includes identity, text, and offering strings. Six-byte JSON escaping fits within the fill budget. Rejected inputs increment the extraction drop counter. Document delivery continues.

Aggregate heap, stream reconstruction, and full request latency still need qualification.

Stream retention

The canonical response-cache buffer limits each stream to 256 KiB of charged input and 1,024 events. Either limit discards all accumulated cache input and prevents further accumulation for that stream. Delivery continues with the original events. EOF, upstream failure, and close release retained input. The byte check precedes cloning and includes media, tool arguments, probability data, metadata, and strings. Copies own their strings so a short substring cannot retain a large provider buffer.

Fixed structure charges and doubled payload charges allow for slice growth and allocation rounding on supported 64-bit hosts. The charge is an admission bound. Full retained-heap and concurrent-stream qualification remain required. EOF response reconstruction and encoding remain synchronous and need separate latency qualification.

Enablement

The master cache switch controls every optional cache kind. Independent flags select chat, embeddings, models, providers, and extractions. See the operator controls. Semantic caching also requires chat caching and its existing opt-ins.

Dedicated shared response cache

The default STARPORT_CACHE_BACKEND=local uses process memory. Select STARPORT_CACHE_BACKEND=valkey with an explicit STARPORT_CACHE_URL and STARPORT_DEPLOYMENT_ID for shared response bytes. Model and extraction caches remain local. Use the same deployment ID for its replicas and different IDs for independent deployments. The default ID is local.

Keys use starport:v1:<base64url-deployment-id>:cache: with unpadded UTF-8 encoding. The admin cache status exposes the effective key_prefix for ACL provisioning.

The URI accepts credentials and an optional database number. Use valkeys:// or rediss:// for TLS with system trust and hostname verification. Plaintext is available on loopback. Remote plaintext requires STARPORT_CACHE_ALLOW_INSECURE=true. Query options, fragments, and additional endpoints are invalid. Redis URI syntax does not establish Redis compatibility qualification.

The cache service must be separate from durable KV. A different database or namespace does not isolate service eviction. Validation rejects the same declared host and port, including common loopback aliases. Operators must also check DNS aliases, proxies, and actual service ownership. The shared cache needs deployment-owned capacity and eviction settings.

Connection work runs in the background with bounded connection attempts and one-second recovery polling. Startup continues while the optional service is unavailable. Reads become misses through the existing proxy error path. Cache failure does not change permission or budget admission.

The connection owner retries initial failures. The client reconnects after connection loss. Shutdown cancels connection work and closes the dedicated client. The admin response_cache.shared status reports the configured service and its availability, without connection values.

Scratch development rejects shared-cache configuration. Use an initialized deployment to test the shared-service recipe. Native TLS deployment, aggregate transport memory, and cache-outage latency remain part of production acceptance.

Shared-cache trust and access

STARPORT_CACHE_CA_FILE selects an explicit PEM trust bundle for a TLS endpoint. The connection uses the bundle's roots. The loader accepts a regular file of at most 1 MiB. File access remains under deployment control. The file appears as cache-ca in the configuration file inventory.

Relative paths require STARPORT_RELATIVE_PATH_BASE=config and resolve under the selected configuration directory. The reader confines the final file lookup to its parent directory. Replace the bundle and restart Starport to change the active trust roots.

Provision a separate cache user for each deployment. Its ACL must restrict keys to the reported key_prefix followed by *. The cache needs these commands: hello, ping, select, client|setname, client|setinfo, get, strlen, set, eval, evalsha, and script|load. The service administrator owns ACL changes. Starport does not grant itself access or claim another namespace.

Replicas of one deployment can share the scoped credential. Independent deployments must use distinct credentials and namespaces.

The connection monitor writes and reads the reserved __starport_health_v1__ key under its namespace. The probe contains one fixed byte and expires after one second. It contains no caller data. A denied probe keeps the optional cache unavailable. It does not block required gateway operations.

The admin response_cache.shared.state field supplies a fixed diagnostic code.

The TLS tests terminate TLS in a local relay before real Valkey traffic. They cover trusted, unknown, expired, and wrong-host certificates. Native service deployments still require CSP15 qualification.