filterapi

package
v0.1.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 20, 2025 License: Apache-2.0 Imports: 3 Imported by: 0

Documentation

Overview

Package filterapi provides the configuration for the AI Gateway-implemented filter which is currently an external processor (See https://github.com/envoyproxy/ai-gateway/issues/90).

This is a public package so that the filter can be testable without depending on the Envoy Gateway as well as it can be used outside the Envoy AI Gateway.

This configuration must be decoupled from the Envoy Gateway types as well as its implementation details. Also, the configuration must not be tied with k8s so it can be tested and iterated without the need for the k8s cluster.

Index

Constants

View Source
const DefaultConfig = `
schema:
  name: OpenAI
selectedBackendHeaderKey: x-ai-eg-selected-backend
modelNameHeaderKey: x-ai-eg-model
`

DefaultConfig is the default configuration that can be used as a fallback when the configuration is not explicitly provided.

Variables

This section is empty.

Functions

This section is empty.

Types

type APIKeyAuth

type APIKeyAuth struct {
	Filename string `json:"filename"`
}

APIKeyAuth defines the file that will be mounted to the external proc.

type APISchemaName

type APISchemaName string

APISchemaName corresponds to APISchemaName in api/v1alpha1/api.go.

const (
	APISchemaOpenAI     APISchemaName = "OpenAI"
	APISchemaAWSBedrock APISchemaName = "AWSBedrock"
)

type AWSAuth

type AWSAuth struct {
	CredentialFileName string `json:"credentialFileName,omitempty"`
	Region             string `json:"region"`
}

AWSAuth defines the credentials needed to access AWS.

type Backend

type Backend struct {
	// Name of the backend, which is the value in the final routing decision
	// matching the header key specified in the [Config.BackendRoutingHeaderKey].
	Name string `json:"name"`
	// Schema specifies the API schema of the output format of requests from.
	Schema VersionedAPISchema `json:"schema"`
	// Weight is the weight of the backend in the routing decision.
	Weight int `json:"weight"`
	// Auth is the authn/z configuration for the backend. Optional.
	// TODO: refactor after https://github.com/envoyproxy/ai-gateway/pull/43.
	Auth *BackendAuth `json:"auth,omitempty"`
}

Backend corresponds to AIGatewayRouteRuleBackendRef in api/v1alpha1/api.go besides that this abstracts the concept of a backend at Envoy Gateway level to a simple name.

type BackendAuth

type BackendAuth struct {
	// APIKey is a location of the api key secret file.
	APIKey *APIKeyAuth `json:"apiKey,omitempty"`
	// AWSAuth specifies the location of the AWS credential file and region.
	AWSAuth *AWSAuth `json:"aws,omitempty"`
}

BackendAuth corresponds partially to BackendSecurityPolicy in api/v1alpha1/api.go.

type Config

type Config struct {
	// UUID is the unique identifier of the filter configuration assigned by the AI Gateway when the configuration is updated.
	UUID string `json:"uuid,omitempty"`
	// MetadataNamespace is the namespace of the dynamic metadata to be used by the filter.
	MetadataNamespace string `json:"metadataNamespace"`
	// LLMRequestCost configures the cost of each LLM-related request. Optional. If this is provided, the filter will populate
	// the "calculated" cost in the filter metadata at the end of the response body processing.
	LLMRequestCosts []LLMRequestCost `json:"llmRequestCosts,omitempty"`
	// InputSchema specifies the API schema of the input format of requests to the filter.
	Schema VersionedAPISchema `json:"schema"`
	// ModelNameHeaderKey is the header key to be populated with the model name by the filter.
	ModelNameHeaderKey string `json:"modelNameHeaderKey"`
	// SelectedBackendHeaderKey is the header key to be populated with the backend name by the filter
	// **after** the routing decision is made by the filter using Rules.
	SelectedBackendHeaderKey string `json:"selectedBackendHeaderKey"`
	// Rules is the routing rules to be used by the filter to make the routing decision.
	// Inside the routing rules, the header ModelNameHeaderKey may be used to make the routing decision.
	Rules []RouteRule `json:"rules"`
}

Config is the configuration schema for the filter.

Example configuration:

schema:
  name: OpenAI
selectedBackendHeaderKey: x-envoy-ai-gateway-selected-backend
modelNameHeaderKey: x-ai-eg-model
llmRequestCosts:
- metadataKey: token_usage_key
  type: OutputToken
rules:
- backends:
  - name: kserve
    weight: 1
    schema:
      name: OpenAI
  - name: awsbedrock
    weight: 10
    schema:
      name: AWSBedrock
  headers:
  - name: x-ai-eg-model
    value: llama3.3333
- backends:
  - name: openai
    schema:
      name: OpenAI
  headers:
  - name: x-ai-eg-model
    value: gpt4.4444

where the input of the Gateway is in the OpenAI schema, the model name is populated in the header x-ai-eg-model, The model name header `x-ai-eg-model` is used in the header matching to make the routing decision. **After** the routing decision is made, the selected backend name is populated in the header `x-ai-eg-selected-backend`. For example, when the model name is `llama3.3333`, the request is routed to either backends `kserve` or `awsbedrock` with weights 1 and 10 respectively, and the selected backend, say `awsbedrock`, is populated in the header `x-ai-eg-selected-backend`.

From Envoy configuration perspective, configuring the header matching based on `x-ai-eg-selected-backend` is enough to route the request to the selected backend. That is because the matching decision is made by the filter and the selected backend is populated in the header `x-ai-eg-selected-backend`.

func MustLoadDefaultConfig

func MustLoadDefaultConfig() (*Config, []byte)

MustLoadDefaultConfig loads the default configuration. This panics if the configuration fails to be loaded.

func UnmarshalConfigYaml

func UnmarshalConfigYaml(path string) (*Config, []byte, error)

UnmarshalConfigYaml reads the file at the given path and unmarshals it into a Config struct.

type HeaderMatch

type HeaderMatch = gwapiv1.HTTPHeaderMatch

HeaderMatch is an alias for HTTPHeaderMatch of the Gateway API.

type LLMRequestCost

type LLMRequestCost struct {
	// MetadataKey is the key of the metadata storing the request cost.
	MetadataKey string `json:"metadataKey"`
	// Type is the kind of the request cost calculation.
	Type LLMRequestCostType `json:"type"`
	// CEL is the CEL expression to calculate the cost of the request.
	// This is not empty when the Type is LLMRequestCostTypeCEL.
	CEL string `json:"cel,omitempty"`
}

LLMRequestCost specifies "where" the request cost is stored in the filter metadata as well as "how" the cost is calculated. By default, the cost is retrieved from "output token" in the response body.

This can be used to subtract the usage token from the usage quota in the rate limit filter when the request completes combined with `apply_on_stream_done` and `hits_addend` fields of the rate limit configuration https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/route/v3/route_components.proto#config-route-v3-ratelimit which is introduced in Envoy 1.33 (to be released soon as of writing).

type LLMRequestCostType

type LLMRequestCostType string

LLMRequestCostType specifies the kind of the request cost calculation.

const (
	// LLMRequestCostTypeOutputToken specifies that the request cost is calculated from the output token.
	LLMRequestCostTypeOutputToken LLMRequestCostType = "OutputToken"
	// LLMRequestCostTypeInputToken specifies that the request cost is calculated from the input token.
	LLMRequestCostTypeInputToken LLMRequestCostType = "InputToken"
	// LLMRequestCostTypeTotalToken specifies that the request cost is calculated from the total token.
	LLMRequestCostTypeTotalToken LLMRequestCostType = "TotalToken"
	// LLMRequestCostTypeCEL specifies that the request cost is calculated from the CEL expression.
	LLMRequestCostTypeCEL LLMRequestCostType = "CEL"
)

type RouteRule

type RouteRule struct {
	// Headers is the list of headers to match for the routing decision.
	// Currently, only exact match is supported.
	Headers []HeaderMatch `json:"headers"`
	// Backends is the list of backends to which the request should be routed to when the headers match.
	Backends []Backend `json:"backends"`
}

RouteRule corresponds to AIGatewayRoute in api/v1alpha1/api.go besides the `Backends` field is modified to abstract the concept of a backend at Envoy Gateway level to a simple name.

type VersionedAPISchema

type VersionedAPISchema struct {
	// Name is the name of the API schema.
	Name APISchemaName `json:"name"`
	// Version is the version of the API schema. Optional.
	Version string `json:"version,omitempty"`
}

VersionedAPISchema corresponds to VersionedAPISchema in api/v1alpha1/api.go.

Directories

Path Synopsis
x
Package x is an experimental package that provides the customizability of the AI Gateway filter.
Package x is an experimental package that provides the customizability of the AI Gateway filter.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL