extproc

package
v0.0.11 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 8, 2026 License: Apache-2.0 Imports: 29 Imported by: 0

README

pkg/envoy/extproc — Envoy ExternalProcessor

This gRPC service implements the Envoy ExternalProcessor protocol. For every inbound HTTP request, it performs the following steps:

  1. Validates the API Key (SecretKeyStore).
  2. Parses the request target (Sandbox ID + Port).
  3. Queries the corresponding Pod IP.
  4. Returns Header mutations to route traffic to the correct Sandbox Pod.

Nothing tells this process about a sandbox

Step 3 reads the Pod informer's sandbox-id index and nothing else. That index is built on a label the claim writes, in the same CAS that moves the Pod from Idle to Starting — so a sandbox is resolvable from the moment it is claimed, well before it is routable, and no notification is needed to make it so. Releasing one strips the same label in one atomic update, so the mapping disappears without anyone being told either.

There was once a cache the Controller pushed into (PushRoute / EvictRoute), to skip the milliseconds between that label write and the watch event arriving here. It is gone. Create now returns only after the sandbox is armed — seconds of round trips, all downstream of the label write — so nothing is waiting in that window any more, and a pushed cache cannot survive replication: one gRPC connection pins to a single backend Pod, so a push reaches one replica and leaves the others disagreeing.

Two consequences worth knowing:

  • This process can be rebuilt and restarted freely. It holds no state the control plane has to re-send. (It once could not: a gate here read an annotation whose writer had been deleted, so any rebuild 502'd every sandbox in the cluster. See docs/troubleshooting/extproc-rebuild-bricks-the-data-plane.md in the deployment repo.)
  • Routing is replica-safe. ActivityTracker is not yet — it is in-memory and per-replica, and the Controller polls one replica for it. That is the remaining blocker for replicas > 1.

Every routing failure is served as 502, including a sandbox that does not exist: this router's only source is an informer cache, so "gone" and "not yet visible" are indistinguishable here and a 404 would assert a permanence it cannot know. The two error values stay distinct in log messages.


File Structure

File Description
server.go gRPC service entry point; contains the ProcessingRequest main loop.
router.go Envoy ExtProc router (bridges gRPC and handlers).
helper.go Core logic: authenticate(), extractTarget(), and RouteTarget.
activity_tracker.go Tracks Sandbox activity (used for idle timeout management).
internal_grpc.go The Controller channel — one RPC, GetLastActive.
helper_test.go Unit tests for extractTarget.

Routing Resolution Strategy (extractTarget)

Strategies are evaluated in order of priority (highest to lowest). The process stops at the first match:

Priority Strategy Format / Source
1 Explicit Headers x-sandbox-id + x-sandbox-port
2 Standard URL Path /sandboxes/<id>/<port>/...

Testing

To run the test suite:

go test ./pkg/envoy/extproc/... -v

Note: If you modify the extractTarget logic, you must update the corresponding test cases in helper_test.go to ensure alignment.

Documentation

Overview

Package extproc implements an Envoy ExternalProcessor (ExtProc) gRPC server.

Architecture:

Client → Envoy :8092 → ExtProc gRPC (this server)
                         ↳ Authenticate API key
                         ↳ Extract Sandbox ID (header or URL path)
                         ↳ Resolve sandbox Pod IP via Kubernetes
                         ↳ Inject "x-envoy-original-dst-host: <PodIP>:<Port>"
                         ↳ Rewrite the :path header if needed
       → Envoy ORIGINAL_DST cluster → Sandbox Pod FastAPI

Index

Constants

This section is empty.

Variables

View Source
var ErrSandboxRouteBadGateway = errors.New("sandbox is not in a healthy state")

ErrSandboxRouteBadGateway signals that the router knows the sandbox exists but cannot route to it right now. Mapped to HTTP 502, retryable. Any Pod state that is not "Running or Stopping + matching sandbox-id label + PodIP set" produces this error, so it also covers informer lag, label race windows, Starting (container swap in progress), Idle/Failed phases, and missing PodIP.

View Source
var ErrSandboxRouteNotFound = errors.New("sandbox route not found")

ErrSandboxRouteNotFound signals that no Pod carries this sandbox ID.

Served as HTTP 502, the same as ErrSandboxRouteBadGateway, and the two are deliberately not distinguished on the wire. This router cannot actually tell "gone" from "not yet visible": its only source is an informer cache, so an index miss means *either* the sandbox never existed *or* this replica has not seen it yet. A 404 would promise a permanence we are not in a position to assert, and the caller's correct move is the same either way — retry, and give up on your own schedule.

The distinction survives where it is useful: the two errors carry different messages, which is what a human reading a log needs.

Functions

This section is empty.

Types

type ActivityFlusher added in v0.0.11

type ActivityFlusher struct {
	// contains filtered or unexported fields
}

ActivityFlusher publishes what this replica has observed into the one place every other component can see: the `last-active` annotation on the sandbox's Pod.

Why it exists. The ActivityTracker is per-replica memory, and with more than one gateway replica no single replica — and therefore no reader — can see all of a sandbox's traffic. The annotation is a rendezvous every replica can write and the controller can read, so each replica contributes what it saw and the union is what comes out.

Why it converges. A replica writes only when the annotation is *older* than what it has observed (so a newer value is never rolled back) and *stale enough to matter* (so a fresh value written by another replica suppresses this one). In steady state that is one write per sandbox per refresh interval for the whole deployment, not one per replica.

The timing contract — how stale is allowed, and therefore how much slack the controller has to add back — lives in pkg/activity, so the writer and the reader cannot drift apart.

func NewActivityFlusher added in v0.0.11

func NewActivityFlusher(tracker *ActivityTracker, k8s client.Client, cfg ActivityFlusherConfig) *ActivityFlusher

NewActivityFlusher builds a flusher. k8s must be the manager's informer-backed client: lookups run per tracked sandbox on every tick.

func (*ActivityFlusher) Run added in v0.0.11

func (f *ActivityFlusher) Run(ctx context.Context)

Run drives the flush loop until ctx is done. Start it only after the informer cache is warm — before that every index lookup misses and the flusher spends its ticks doing nothing.

type ActivityFlusherConfig added in v0.0.11

type ActivityFlusherConfig struct {
	// Tick is how often the flusher looks for work. Zero means activity.FlushTick.
	Tick time.Duration

	// RateLimitPerSecond caps the Pod PATCH rate. Zero means unlimited, which is
	// the shipped default: the right number depends on how many sandboxes this
	// deployment actually keeps busy, and a limit set below the steady-state
	// demand makes the queue grow without bound — the annotations then go stale
	// and the controller releases sandboxes that are in use. Measure first
	// (agentbox_gateway_activity_flush_total), then set it above the observed
	// peak. The limiter smooths bursts; it is not a way to save API server load.
	RateLimitPerSecond int

	// Workers is how many PATCHes run concurrently. Zero means 4.
	Workers int

	// MaxPending bounds the deferred-write map. Beyond it the oldest entries are
	// dropped and counted, because an unbounded map is a memory leak and a
	// silently growing queue is worse than a counted loss.
	MaxPending int

	// WriteTimeout bounds a single PATCH. Zero means 5s.
	WriteTimeout time.Duration
}

ActivityFlusherConfig tunes the flusher. The zero value is the production configuration.

type ActivityTracker

type ActivityTracker struct {
	// contains filtered or unexported fields
}

ActivityTracker is a lightweight in-memory store that records the last time a proxied HTTP request was observed for each sandbox, by *this* replica. It is intentionally stateless with respect to Kubernetes: it holds no Pod data and does no I/O.

It is not the whole truth, and it is not meant to be. A sandbox's traffic can land on any gateway replica, so this map is one replica's view; what makes it useful is that every replica publishes its view into the same place — the sandbox Pod's last-active annotation — from which the controller reads the union. See ActivityFlusher.

Lifecycle:

  1. On startup, InitFromAnnotations is called for each Running pod that has a last-active annotation so that ExtProc restart does not reset history.
  2. On every proxied request, Touch is called (non-blocking, O(1)).
  3. ActivityFlusher snapshots the map on a ticker and writes what this replica saw into the Pod annotation.

func NewActivityTracker

func NewActivityTracker() *ActivityTracker

NewActivityTracker creates an ActivityTracker with an empty state and no background GC. This is the zero-config constructor.

func NewActivityTrackerWithGC

func NewActivityTrackerWithGC(c client.Client, gcInterval time.Duration) *ActivityTracker

NewActivityTrackerWithGC creates an ActivityTracker with a background GC goroutine that periodically removes entries for sandboxes that are no longer in the Running state. gcInterval controls the GC frequency; 5–10 minutes is recommended for production use.

func (*ActivityTracker) InitFromAnnotations

func (t *ActivityTracker) InitFromAnnotations(sandboxID string, ts time.Time)

InitFromAnnotations seeds the in-memory map with a timestamp from a pod annotation. Call this once per Running pod at startup before serving traffic. If ts is the zero value it is ignored.

func (*ActivityTracker) Remove

func (t *ActivityTracker) Remove(sandboxID string)

Remove deletes a sandbox entry from the tracker. Called when a pod is no longer Running so its entry does not accumulate indefinitely.

func (*ActivityTracker) StartGC

func (t *ActivityTracker) StartGC(ctx context.Context)

StartGC launches the background GC goroutine. It is a no-op when no client or gcInterval was provided. Call this only after the K8s cache is warm so that the first GC cycle does not spuriously evict entries.

func (*ActivityTracker) Touch

func (t *ActivityTracker) Touch(sandboxID string)

Touch records the current time as the last-active timestamp for sandboxID. It is safe to call from multiple goroutines and returns immediately.

type K8sSandboxRouter

type K8sSandboxRouter struct {
	// contains filtered or unexported fields
}

K8sSandboxRouter resolves sandbox routes from one data source: the Pod informer, queried through the sandbox-id index.

Why nobody tells the gateway about a sandbox

The mapping this router needs — sandbox ID to Pod — is a label the claim writes onto the Pod in the same CAS that moves it Idle → Starting. The indexer is built on that label, so every replica of this process can resolve a sandbox from the moment it is claimed, without being told. There was once a cache the control plane pushed into, to skip the milliseconds between that write and the watch event arriving here. It is gone, for two reasons:

  • Nothing is waiting on those milliseconds any more. Create returns only after the sandbox is armed — runtimes answering, env vars delivered, CA installed, egress policy and credentials pushed — which is seconds of round trips, all of it after the label write this router keys on. By the time a caller holds an ID, the informer has long since caught up.
  • A pushed cache cannot survive replication. The control plane holds one gRPC connection, which pins to a single backend Pod, so a push reaches one replica and the others answer differently for the same sandbox. An informer needs no such coordination: each replica watches the apiserver and arrives at the same answer on its own.

So this process is a pure reader of Kubernetes state. It can be restarted, rebuilt, or scaled to any number of replicas with no control-plane involvement at all.

func NewK8sSandboxRouter

func NewK8sSandboxRouter(c client.Client, defaultPort int) *K8sSandboxRouter

NewK8sSandboxRouter creates a K8sSandboxRouter. The defaultPort parameter is kept for compatibility but is not used when the caller provides a port via headers or URL.

func (*K8sSandboxRouter) ResolveSandboxRoute

func (r *K8sSandboxRouter) ResolveSandboxRoute(ctx context.Context, sandboxID string, reqPort int) (*SandboxRoute, error)

ResolveSandboxRoute returns the live Pod IP for the given sandbox, or one of ErrSandboxRouteNotFound / ErrSandboxRouteBadGateway. Exactly one Pod read is performed per request, served from the informer cache.

type RouteTarget

type RouteTarget struct {
	SandboxID     string
	Port          int
	RewrittenPath string
}

RouteTarget holds the extracted sandbox ID, port, and the rewritten path.

type SandboxRoute

type SandboxRoute struct {
	PodIP string
	Port  int
}

SandboxRoute holds the resolved upstream destination for a sandbox request.

func (*SandboxRoute) DestHost

func (r *SandboxRoute) DestHost() string

DestHost formats the route as "IP:Port" for the x-envoy-original-dst-host header.

type SandboxRouter

type SandboxRouter interface {
	ResolveSandboxRoute(ctx context.Context, sandboxID string, port int) (*SandboxRoute, error)
}

SandboxRouter resolves the destination Pod for a given sandbox ID. Implement this interface to swap in a cache-backed or test-double router.

type Server

type Server struct {
	extProcPb.UnimplementedExternalProcessorServer
	// contains filtered or unexported fields
}

Server is the gRPC ExtProc server that runs alongside the Gin REST API.

func New

func New(cfg ServerConfig, adminKeyMgr *apikey.AdminKeyManager, keyStore apikey.KeyStore, router SandboxRouter, activityTracker *ActivityTracker, clusterStore *cluster.Store, localClusterID string) *Server

New creates a new ExtProc gRPC Server.

  • bindAddr – TCP address to listen on (e.g. ":9002").
  • adminKeyMgr – Admin key manager (pass nil to disable auth / dev mode).
  • keyStore – Tenant key store for validating opaque tokens.
  • router – Resolves sandbox ID → Pod IP:Port.
  • activityTracker – Records per-sandbox last-active timestamps (may be nil).
  • clusterStore – Cross-cluster config store (nil to disable cross-cluster routing).
  • localClusterID – Identifier of the local cluster (empty to disable cross-cluster routing).

func (*Server) Process

Process implements envoy.service.ext_proc.v3.ExternalProcessor/Process.

The method runs a receive loop over the bidirectional stream. In the current configuration Envoy only sends RequestHeaders messages (all body/trailer modes are set to NONE/SKIP), so one exchange per HTTP request is expected.

func (*Server) Start

func (s *Server) Start(ctx context.Context) error

Start listens and serves until ctx is cancelled. It is designed to be run in a goroutine alongside the controller-manager and Gin API server.

type ServerConfig

type ServerConfig struct {
	BindAddr   string
	EnableAuth bool
}

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL