Documentation
¶
Overview ¶
Package extproc implements an Envoy ExternalProcessor (ExtProc) gRPC server.
Architecture:
Client → Envoy :8092 → ExtProc gRPC (this server)
↳ Authenticate API key
↳ Extract Sandbox ID (header or URL path)
↳ Resolve sandbox Pod IP via Kubernetes
↳ Inject "x-envoy-original-dst-host: <PodIP>:<Port>"
↳ Rewrite the :path header if needed
→ Envoy ORIGINAL_DST cluster → Sandbox Pod FastAPI
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrSandboxRouteBadGateway = errors.New("sandbox is not in a healthy state")
ErrSandboxRouteBadGateway signals that the router knows the sandbox exists but cannot route to it right now. Mapped to HTTP 502, retryable. Any Pod state that is not "Running or Stopping + matching sandbox-id label + PodIP set" produces this error, so it also covers informer lag, label race windows, Starting (container swap in progress), Idle/Failed phases, and missing PodIP.
var ErrSandboxRouteNotFound = errors.New("sandbox route not found")
ErrSandboxRouteNotFound signals that no Pod carries this sandbox ID.
Served as HTTP 502, the same as ErrSandboxRouteBadGateway, and the two are deliberately not distinguished on the wire. This router cannot actually tell "gone" from "not yet visible": its only source is an informer cache, so an index miss means *either* the sandbox never existed *or* this replica has not seen it yet. A 404 would promise a permanence we are not in a position to assert, and the caller's correct move is the same either way — retry, and give up on your own schedule.
The distinction survives where it is useful: the two errors carry different messages, which is what a human reading a log needs.
Functions ¶
This section is empty.
Types ¶
type ActivityFlusher ¶ added in v0.0.11
type ActivityFlusher struct {
// contains filtered or unexported fields
}
ActivityFlusher publishes what this replica has observed into the one place every other component can see: the `last-active` annotation on the sandbox's Pod.
Why it exists. The ActivityTracker is per-replica memory, and with more than one gateway replica no single replica — and therefore no reader — can see all of a sandbox's traffic. The annotation is a rendezvous every replica can write and the controller can read, so each replica contributes what it saw and the union is what comes out.
Why it converges. A replica writes only when the annotation is *older* than what it has observed (so a newer value is never rolled back) and *stale enough to matter* (so a fresh value written by another replica suppresses this one). In steady state that is one write per sandbox per refresh interval for the whole deployment, not one per replica.
The timing contract — how stale is allowed, and therefore how much slack the controller has to add back — lives in pkg/activity, so the writer and the reader cannot drift apart.
func NewActivityFlusher ¶ added in v0.0.11
func NewActivityFlusher(tracker *ActivityTracker, k8s client.Client, cfg ActivityFlusherConfig) *ActivityFlusher
NewActivityFlusher builds a flusher. k8s must be the manager's informer-backed client: lookups run per tracked sandbox on every tick.
func (*ActivityFlusher) Run ¶ added in v0.0.11
func (f *ActivityFlusher) Run(ctx context.Context)
Run drives the flush loop until ctx is done. Start it only after the informer cache is warm — before that every index lookup misses and the flusher spends its ticks doing nothing.
type ActivityFlusherConfig ¶ added in v0.0.11
type ActivityFlusherConfig struct {
// Tick is how often the flusher looks for work. Zero means activity.FlushTick.
Tick time.Duration
// RateLimitPerSecond caps the Pod PATCH rate. Zero means unlimited, which is
// the shipped default: the right number depends on how many sandboxes this
// deployment actually keeps busy, and a limit set below the steady-state
// demand makes the queue grow without bound — the annotations then go stale
// and the controller releases sandboxes that are in use. Measure first
// (agentbox_gateway_activity_flush_total), then set it above the observed
// peak. The limiter smooths bursts; it is not a way to save API server load.
RateLimitPerSecond int
// Workers is how many PATCHes run concurrently. Zero means 4.
Workers int
// MaxPending bounds the deferred-write map. Beyond it the oldest entries are
// dropped and counted, because an unbounded map is a memory leak and a
// silently growing queue is worse than a counted loss.
MaxPending int
// WriteTimeout bounds a single PATCH. Zero means 5s.
WriteTimeout time.Duration
}
ActivityFlusherConfig tunes the flusher. The zero value is the production configuration.
type ActivityTracker ¶
type ActivityTracker struct {
// contains filtered or unexported fields
}
ActivityTracker is a lightweight in-memory store that records the last time a proxied HTTP request was observed for each sandbox, by *this* replica. It is intentionally stateless with respect to Kubernetes: it holds no Pod data and does no I/O.
It is not the whole truth, and it is not meant to be. A sandbox's traffic can land on any gateway replica, so this map is one replica's view; what makes it useful is that every replica publishes its view into the same place — the sandbox Pod's last-active annotation — from which the controller reads the union. See ActivityFlusher.
Lifecycle:
- On startup, InitFromAnnotations is called for each Running pod that has a last-active annotation so that ExtProc restart does not reset history.
- On every proxied request, Touch is called (non-blocking, O(1)).
- ActivityFlusher snapshots the map on a ticker and writes what this replica saw into the Pod annotation.
func NewActivityTracker ¶
func NewActivityTracker() *ActivityTracker
NewActivityTracker creates an ActivityTracker with an empty state and no background GC. This is the zero-config constructor.
func NewActivityTrackerWithGC ¶
func NewActivityTrackerWithGC(c client.Client, gcInterval time.Duration) *ActivityTracker
NewActivityTrackerWithGC creates an ActivityTracker with a background GC goroutine that periodically removes entries for sandboxes that are no longer in the Running state. gcInterval controls the GC frequency; 5–10 minutes is recommended for production use.
func (*ActivityTracker) InitFromAnnotations ¶
func (t *ActivityTracker) InitFromAnnotations(sandboxID string, ts time.Time)
InitFromAnnotations seeds the in-memory map with a timestamp from a pod annotation. Call this once per Running pod at startup before serving traffic. If ts is the zero value it is ignored.
func (*ActivityTracker) Remove ¶
func (t *ActivityTracker) Remove(sandboxID string)
Remove deletes a sandbox entry from the tracker. Called when a pod is no longer Running so its entry does not accumulate indefinitely.
func (*ActivityTracker) StartGC ¶
func (t *ActivityTracker) StartGC(ctx context.Context)
StartGC launches the background GC goroutine. It is a no-op when no client or gcInterval was provided. Call this only after the K8s cache is warm so that the first GC cycle does not spuriously evict entries.
func (*ActivityTracker) Touch ¶
func (t *ActivityTracker) Touch(sandboxID string)
Touch records the current time as the last-active timestamp for sandboxID. It is safe to call from multiple goroutines and returns immediately.
type K8sSandboxRouter ¶
type K8sSandboxRouter struct {
// contains filtered or unexported fields
}
K8sSandboxRouter resolves sandbox routes from one data source: the Pod informer, queried through the sandbox-id index.
Why nobody tells the gateway about a sandbox ¶
The mapping this router needs — sandbox ID to Pod — is a label the claim writes onto the Pod in the same CAS that moves it Idle → Starting. The indexer is built on that label, so every replica of this process can resolve a sandbox from the moment it is claimed, without being told. There was once a cache the control plane pushed into, to skip the milliseconds between that write and the watch event arriving here. It is gone, for two reasons:
- Nothing is waiting on those milliseconds any more. Create returns only after the sandbox is armed — runtimes answering, env vars delivered, CA installed, egress policy and credentials pushed — which is seconds of round trips, all of it after the label write this router keys on. By the time a caller holds an ID, the informer has long since caught up.
- A pushed cache cannot survive replication. The control plane holds one gRPC connection, which pins to a single backend Pod, so a push reaches one replica and the others answer differently for the same sandbox. An informer needs no such coordination: each replica watches the apiserver and arrives at the same answer on its own.
So this process is a pure reader of Kubernetes state. It can be restarted, rebuilt, or scaled to any number of replicas with no control-plane involvement at all.
func NewK8sSandboxRouter ¶
func NewK8sSandboxRouter(c client.Client, defaultPort int) *K8sSandboxRouter
NewK8sSandboxRouter creates a K8sSandboxRouter. The defaultPort parameter is kept for compatibility but is not used when the caller provides a port via headers or URL.
func (*K8sSandboxRouter) ResolveSandboxRoute ¶
func (r *K8sSandboxRouter) ResolveSandboxRoute(ctx context.Context, sandboxID string, reqPort int) (*SandboxRoute, error)
ResolveSandboxRoute returns the live Pod IP for the given sandbox, or one of ErrSandboxRouteNotFound / ErrSandboxRouteBadGateway. Exactly one Pod read is performed per request, served from the informer cache.
type RouteTarget ¶
RouteTarget holds the extracted sandbox ID, port, and the rewritten path.
type SandboxRoute ¶
SandboxRoute holds the resolved upstream destination for a sandbox request.
func (*SandboxRoute) DestHost ¶
func (r *SandboxRoute) DestHost() string
DestHost formats the route as "IP:Port" for the x-envoy-original-dst-host header.
type SandboxRouter ¶
type SandboxRouter interface {
ResolveSandboxRoute(ctx context.Context, sandboxID string, port int) (*SandboxRoute, error)
}
SandboxRouter resolves the destination Pod for a given sandbox ID. Implement this interface to swap in a cache-backed or test-double router.
type Server ¶
type Server struct {
extProcPb.UnimplementedExternalProcessorServer
// contains filtered or unexported fields
}
Server is the gRPC ExtProc server that runs alongside the Gin REST API.
func New ¶
func New(cfg ServerConfig, adminKeyMgr *apikey.AdminKeyManager, keyStore apikey.KeyStore, router SandboxRouter, activityTracker *ActivityTracker, clusterStore *cluster.Store, localClusterID string) *Server
New creates a new ExtProc gRPC Server.
- bindAddr – TCP address to listen on (e.g. ":9002").
- adminKeyMgr – Admin key manager (pass nil to disable auth / dev mode).
- keyStore – Tenant key store for validating opaque tokens.
- router – Resolves sandbox ID → Pod IP:Port.
- activityTracker – Records per-sandbox last-active timestamps (may be nil).
- clusterStore – Cross-cluster config store (nil to disable cross-cluster routing).
- localClusterID – Identifier of the local cluster (empty to disable cross-cluster routing).
func (*Server) Process ¶
func (s *Server) Process(srv extProcPb.ExternalProcessor_ProcessServer) error
Process implements envoy.service.ext_proc.v3.ExternalProcessor/Process.
The method runs a receive loop over the bidirectional stream. In the current configuration Envoy only sends RequestHeaders messages (all body/trailer modes are set to NONE/SKIP), so one exchange per HTTP request is expected.