extproc

package
v0.0.0-...-a0319ed Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 2, 2026 License: AGPL-3.0 Imports: 52 Imported by: 0

Documentation

Overview

Network ext_proc server. Implements the L4 sibling of the HTTP ext_proc server in server.go. Wired onto the same gRPC endpoint by RegisterNetworkServer; reuses sinkRegistry for OTLP fanout. Receives agent → server bytes (ProcessingMode.ProcessRead = STREAMED), counts + discards them, and emits one L4Record per connection on stream close. Server → agent bytes are unobserved (ProcessWrite = SKIP).

Package extproc implements envoy.service.ext_proc.v3.ExternalProcessor. Streams through an Envoy ext_proc filter deliver HTTP headers + body chunks (both directions) to this server; we buffer them per stream, emit a structured log on stream close, and hand back continue responses that leave traffic untouched.

The EG extension (see internal/egextension) wires Envoy so that MITM listener traffic flows through the filter, and so that PROXY v2 TLVs (agent identity + invocation ID) appear as dynamic metadata under the clrk.apoxy.dev namespace — which we read from MetadataContext in the downstream handler.

One gRPC service serves two filter positions. Downstream (listener- level) streams carry the agent-side transaction and are handled by downstreamStream (downstream.go). Upstream (cluster-level) streams — one per router attempt against a synthesized LLM cluster — are stamped with x-clrk-extproc-role: upstream in the filter's GrpcService initial_metadata and get the per-attempt handler.

Index

Constants

View Source
const (
	MetaAgentKind      = "agent_kind"
	MetaAgentNamespace = "agent_namespace"
	MetaAgentName      = "agent_name"
	MetaAgentUID       = "agent_uid"
	MetaAgentRevision  = "agent_revision"
	MetaInvocationID   = "invocation_id"

	// MetaDstName carries the DNS-bound destination hostname for the
	// connection — the name the sandbox's resolver answered for the
	// dst IP. Worker emits it via PROXY v2 TLVDstName; the listener
	// filter publishes it under this key. Empty when no binding
	// existed at dial time (direct-IP, DNS bypass, expired entry).
	MetaDstName = "dst_name"
)

Metadata sub-keys under MetadataNamespace.

View Source
const MetadataNamespace = "clrk.apoxy.dev"

MetadataNamespace is the dynamic-metadata key carrying clrk PROXY v2 TLV values decoded by the proxy_protocol listener filter. Keep in sync with internal/egextension when it programs listener-filter rules.

Variables

This section is empty.

Functions

func CredPoliciesVersion

CredPoliciesVersion fingerprints the CIPs that attach to the given EG (directly or via an APR). Mirrors aiproviderRoutesVersion in shape: sorted parts list, one per matched CIP, encoding the CIP's own ResourceVersion AND the ResourceVersion of its referenced Secret. Used by sinkRegistry to decide whether to rebuild the cached egSink.

Including the Secret's RV is what makes credential rotation propagate without operator intervention: a Secret rotation bumps the underlying RV, the next ext_proc stream computes a different fingerprint, the sink rebuilds, and resolveCIP reads the fresh value. Secrets that don't exist yet (or vanished) get a `none` sentinel so the fingerprint changes again when the Secret (re)appears.

Per-stream cost: one cached-client Get per matched CIP. The client is informer-backed so Gets are in-memory cache hits — no extra apiserver traffic.

func RegisterNetworkServer

func RegisterNetworkServer(s *grpc.Server, server *Server)

RegisterNetworkServer attaches the L4 ext_proc service to the given gRPC server. Call alongside the HTTP ExternalProcessorServer registration in cmd/controller-manager so a single gRPC listener serves both.

Types

type L4Record

type L4Record struct {
	Timestamp time.Time
	EndAt     time.Time

	// Agent identity, populated from the clrk.apoxy.dev dynamic
	// metadata namespace forwarded by Envoy on the first
	// ProcessingRequest (same shape the HTTP path reads).
	AgentKind      string
	AgentNamespace string
	AgentName      string
	AgentUID       string
	AgentRevision  string
	InvocationID   string

	// DstName is the DNS-bound destination hostname the sandbox's
	// resolver answered for the connection's destination IP. Carried
	// from the worker via PROXY v2 TLVDstName and forwarded by Envoy
	// as MetaDstName under the clrk.apoxy.dev namespace. Empty when
	// no binding existed at dial time (direct-IP, DNS bypass,
	// expired entry, or DoT/DoH bypass).
	DstName string

	// BytesUpstream is the count of agent → server bytes observed on
	// this connection. Server → agent bytes are unobserved by design.
	BytesUpstream int64
}

L4Record is the structured representation of one TCP/TLS connection observed through the network ext_proc filter on the egress listener's TCP-fallback chain. Distinct from Record (HTTP) because at L4 there are no headers, no body, and no GenAI parser hooks — just connection-level facts.

Bytes counted are the read direction only: ProcessingMode is STREAMED on read and SKIP on write, which means we observe agent → server bytes (the audit direction) and stay out of the server → agent path. Connection duration is wall-clock between the first ReadData event and stream close.

type NetworkServer

type NetworkServer struct {
	netextprocsvcv3.UnimplementedNetworkExternalProcessorServer
	// contains filtered or unexported fields
}

NetworkServer implements the network_ext_proc.v3 NetworkExternalProcessor service. One gRPC stream corresponds to one TCP/TLS connection on the egress listener's TCP-fallback chain.

func NewNetworkServer

func NewNetworkServer(s *Server) *NetworkServer

NewNetworkServer constructs a network ext_proc server that emits to the same per-EG sinks the HTTP server uses.

func (*NetworkServer) Process

Process handles one network ext_proc stream. With ProcessRead = STREAMED, ProcessWrite = SKIP the filter sends ReadData chunks for every agent → server packet plus a final ReadData{end_of_stream:true} when the connection closes. We respond UNMODIFIED + CONTINUE on every chunk so Envoy forwards the original bytes unchanged.

type Record

type Record struct {
	Timestamp time.Time
	EndAt     time.Time

	AgentKind      string
	AgentNamespace string
	AgentName      string
	AgentUID       string
	AgentRevision  string
	InvocationID   string

	RequestHeadersAt time.Time
	RequestHeaders   map[string]string
	RequestBodyAt    time.Time
	RequestBody      []byte
	RequestTruncated bool

	ResponseHeadersAt time.Time
	ResponseHeaders   map[string]string
	ResponseBodyAt    time.Time
	ResponseBody      []byte
	ResponseTruncated bool

	// RequestBodyDecoded / ResponseBodyDecoded carry the content-encoding-
	// decoded copy of the body when enrichRecord inflated it (agents send
	// `accept-encoding: gzip, ...` and the MITM forwards it, so providers
	// gzip/br/zstd even SSE bodies). The OTLP body span event ships these
	// so the inspector renders readable JSON instead of a compressed blob.
	// Nil when the body was identity-encoded, truncated (a header-less
	// keep-last-N tail no codec can inflate), or failed to inflate -- the
	// sink then falls back to the raw RequestBody/ResponseBody. The raw
	// bytes are never overwritten, so clrk.req.bytes / clrk.resp.bytes stay
	// true on-wire counts.
	//
	// RequestContentEncoding / ResponseContentEncoding name the non-identity
	// wire encoding that was present (e.g. "gzip"), set whenever the body
	// carried one regardless of whether it inflated, so the sink can label a
	// decoded body as "was gzipped" and a truncated/undecodable compressed
	// body as not decodable.
	RequestBodyDecoded      []byte
	RequestContentEncoding  string
	ResponseBodyDecoded     []byte
	ResponseContentEncoding string
	// ResponseBodyChunks counts ResponseBody ProcessingRequest messages
	// received from Envoy. Under ResponseBodyMode=BUFFERED this is 1
	// (or 0 if the upstream produced no body); under STREAMED it
	// reflects per-chunk delivery. Surfaced on OTLP so operators (and
	// integration tests) can confirm the conditional STREAMED override
	// promoted the mode for streaming traffic.
	ResponseBodyChunks int

	// RequestBodyRewritten is true when ext_proc replaced the request
	// body before forwarding (e.g. forcing OpenAI
	// stream_options.include_usage=true so streamed responses always
	// emit terminal usage). Surfaced on the OTLP record as
	// clrk.body.request_rewritten so operators can correlate
	// upstream-observed bodies with what the agent actually sent.
	RequestBodyRewritten bool

	// Provider holds parsed AI-provider facts (gen_ai.* shape) when the
	// request's :authority host matched a known provider. Nil for any
	// other host.
	Provider *parsers.ProviderInfo

	// MatchedRouteNamespace / MatchedRouteName identify the
	// AIProviderRoute that accepted this transaction; empty when no
	// route attached to the calling EG matched.
	MatchedRouteNamespace string
	MatchedRouteName      string

	// BudgetDenied is set when the matched route's TokenBudget caused
	// us to short-circuit the request with an HTTP 429. The captured
	// record will only have RequestHeaders populated in that case
	// (response phases never ran).
	BudgetDenied bool
	// BudgetDailyUsed / BudgetDailyMax are the counter snapshot at the
	// moment of the deny decision, attached for operator visibility.
	BudgetDailyUsed int64
	BudgetDailyMax  int64

	// MCP carries parsed MCP JSON-RPC envelope facts when the
	// :authority host fell under an attached MCPRoute and the request
	// body was a single (non-batch) JSON-RPC request. Nil for any
	// other traffic.
	MCP *parsers.MCPInfo

	// MatchedMCPRouteNamespace / MatchedMCPRouteName identify the
	// MCPRoute whose rule accepted this transaction; empty when no
	// rule matched.
	MatchedMCPRouteNamespace string
	MatchedMCPRouteName      string

	// MCPToolPolicyDecision is "allow" or "deny" when the matched
	// rule's ToolPolicy fired on a tools/call request; empty for
	// non-tools/call methods, no-policy rules, or unmatched traffic.
	MCPToolPolicyDecision string

	// SelectedBackendNamespace / SelectedBackendName identify the clrk
	// Backend this transaction was re-pointed to at RequestBody EOS.
	// SelectedBackendSchema is that backend's canonical gen_ai.system
	// (the wire schema the response was parsed against).
	// SelectedBackendReselected is true only when re-selection actually
	// fired. All four stay zero for single-backend / non-reselectable
	// routes, so OTLP attributes are unchanged for existing traffic.
	SelectedBackendNamespace  string
	SelectedBackendName       string
	SelectedBackendSchema     string
	SelectedBackendReselected bool

	// Cross-schema translation facts (APO-742). TranslationApplied is
	// true when the request committed upstream in a different wire
	// schema than the agent spoke; From/To are the canonical
	// gen_ai.system names of those schemas. SkippedBackends counts
	// cross-schema candidates dropped by per-request filtering
	// (streaming, capability misses, no applicable modelRewrite).
	// DroppedExtras counts source-schema members not representable in
	// the target schema (request + response sides). Error carries the
	// reason translation degraded: a fallback's decode/encode failure,
	// or the response-side failure behind a fail-closed 502.
	TranslationApplied         bool
	TranslationFrom            string
	TranslationTo              string
	TranslationSkippedBackends int
	TranslationDroppedExtras   int
	TranslationError           string

	// Attempts is the number of upstream router attempts that recorded
	// facts for this request; AttemptBackends is the ordered
	// "<ns>/<name>" walk of the backends they targeted (the last entry
	// is the serving attempt, mirrored in SelectedBackend*). Both stay
	// zero for unpinned traffic.
	Attempts        int
	AttemptBackends []string

	// RetryIneligibleReason is set when a pinned request cannot be
	// retried — today only "body_too_large": the request body exceeded
	// the synthesized route's retry buffer, so Envoy silently disables
	// retries and an attached FallbackRoutingPolicy cannot fire.
	RetryIneligibleReason string
}

Record is the structured representation of one HTTP request/response observed through ext_proc. All fields are best-effort; Envoy may not deliver bodies or trailers depending on mode.

Timestamps mark the wall-clock arrival of each ext_proc callback so the trace sink can synthesize span events with realistic timings. They are zero when the corresponding phase did not occur (e.g. an empty-body request leaves RequestBodyAt zero).

func (Record) RequestBodyForEvent

func (r Record) RequestBodyForEvent() []byte

RequestBodyForEvent returns the bytes to ship in the OTLP request body span event: the content-decoded copy when enrichRecord inflated it, else the raw captured bytes.

func (Record) ResponseBodyForEvent

func (r Record) ResponseBodyForEvent() []byte

ResponseBodyForEvent is RequestBodyForEvent for the response direction.

type Server

type Server struct {
	extprocv3.UnimplementedExternalProcessorServer
	// contains filtered or unexported fields
}

Server implements ExternalProcessorServer.

func New

func New(c client.Client, opts ...ServerOption) *Server

New constructs an ext_proc server. The client is used to look up EgressGateway state per-stream (OTLP endpoint + body capture bounds); pass cm.GetClient() in the controller-manager. Tests may pass nil and use WithSinkOverride.

func (*Server) Process

Process handles one ext_proc stream. A downstream stream corresponds to one HTTP transaction (request + response pair, assuming upstream keep-alives); an upstream stream corresponds to one router attempt.

func (*Server) Stop

func (s *Server) Stop(ctx context.Context)

Stop releases any resources held by per-EgressGateway sinks (in particular, OTLP exporter background workers and pending batches). Safe to call multiple times.

type ServerOption

type ServerOption func(*Server)

ServerOption configures the ext_proc Server.

func WithInvocationContext

func WithInvocationContext(store *invocationctx.Store) ServerOption

WithInvocationContext threads the controller-manager-local store that carries an invocation's W3C trace parent context from ingress to egress. The egress filter consults it on every stream's RequestHeaders phase to (a) parent the egress span on the inbound trace and (b) inject `traceparent` on the outbound request when the agent didn't set one — see the package doc on internal/extproc/invocationctx.

func WithSinkOverride

func WithSinkOverride(sink Sink) ServerOption

WithSinkOverride forces every stream to use the given sink, bypassing per-EgressGateway lookup. Intended for tests; production wires the controller-runtime client and resolves sinks per-EG.

type Sink

type Sink interface {
	Emit(record Record)
	EmitL4(record L4Record)
}

Sink receives one captured record per HTTP transaction or L4 connection. Implementations might push to OTLP, Axiom, stdout, etc. Errors are logged and dropped.

Directories

Path Synopsis
Package awssigv4 implements AWS Signature Version 4 request signing for credential injection at the egress proxy.
Package awssigv4 implements AWS Signature Version 4 request signing for credential injection at the egress proxy.
Package capture holds the body-capture and HTTP request-line primitives shared by the egress sink (internal/extproc) and the ingress dispatch span (internal/extproc/ingress).
Package capture holds the body-capture and HTTP request-line primitives shared by the egress sink (internal/extproc) and the ingress dispatch span (internal/extproc/ingress).
Package ingress implements the TaskAgent ingress ext_proc filter.
Package ingress implements the TaskAgent ingress ext_proc filter.
Package invocationctx is the controller-manager-local map that carries a TaskAgent invocation's W3C trace parent context from the ingress edge to every outbound LLM/MCP call the agent makes.
Package invocationctx is the controller-manager-local map that carries a TaskAgent invocation's W3C trace parent context from the ingress edge to every outbound LLM/MCP call the agent makes.
Package jsonx pins the egress data path to a single JSON engine: bytedance/sonic in std-compatible mode.
Package jsonx pins the egress data path to a single JSON engine: bytedance/sonic in std-compatible mode.
Package llmcall defines clrk's canonical intermediate representation (IR) for LLM API traffic and the provider registry that converts wire schemas (Anthropic, OpenAI, Google, ...) to and from it.
Package llmcall defines clrk's canonical intermediate representation (IR) for LLM API traffic and the provider registry that converts wire schemas (Anthropic, OpenAI, Google, ...) to and from it.
providers/all
Package all registers every built-in llmcall provider via blank imports.
Package all registers every built-in llmcall provider via blank imports.
providers/anthropic
Package anthropic is the llmcall provider plugin for api.anthropic.com (the Anthropic Messages API).
Package anthropic is the llmcall provider plugin for api.anthropic.com (the Anthropic Messages API).
providers/azureopenai
Package azureopenai is the llmcall provider plugin for Azure OpenAI (*.openai.azure.com).
Package azureopenai is the llmcall provider plugin for Azure OpenAI (*.openai.azure.com).
providers/bedrock
Package bedrock is the llmcall provider plugin for AWS Bedrock's Converse API (bedrock-runtime.<region>.amazonaws.com).
Package bedrock is the llmcall provider plugin for AWS Bedrock's Converse API (bedrock-runtime.<region>.amazonaws.com).
providers/google
Package google is the llmcall provider plugin for generativelanguage.googleapis.com (Google's consumer Gemini API; "google_genai" in OTel gen_ai semconv).
Package google is the llmcall provider plugin for generativelanguage.googleapis.com (Google's consumer Gemini API; "google_genai" in OTel gen_ai semconv).
providers/openai
Package openai is the llmcall provider plugin for api.openai.com (the OpenAI Chat Completions API surface).
Package openai is the llmcall provider plugin for api.openai.com (the OpenAI Chat Completions API surface).
Package parsers extracts AI-provider-specific facts from buffered HTTP request/response pairs captured by ext_proc.
Package parsers extracts AI-provider-specific facts from buffered HTTP request/response pairs captured by ext_proc.
Package tracectx is the shared W3C trace-context utility used by both the ingress and egress ext_proc filters: it owns the singleton TextMapPropagator, the sensitive-header redaction list, and the helpers that extract a parent context from inbound headers / build a child traceparent for outbound injection.
Package tracectx is the shared W3C trace-context utility used by both the ingress and egress ext_proc filters: it owns the singleton TextMapPropagator, the sensitive-header redaction list, and the helpers that extract a parent context from inbound headers / build a child traceparent for outbound injection.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL