Documentation
¶
Overview ¶
Package x is an experimental package that provides the customizability of the AI Gateway filter.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ErrNoMatchingRule = errors.New("no matching rule found")
ErrNoMatchingRule is the error the router function must return if there is no matching rule.
Functions ¶
This section is empty.
Types ¶
type ChatCompletionMetrics ¶ added in v0.2.0
type ChatCompletionMetrics interface {
// StartRequest initializes timing for a new request.
StartRequest(headers map[string]string)
// SetModel sets the model the request. This is usually called after parsing the request body .
SetModel(model string)
// SetBackend sets the selected backend when the routing decision has been made. This is usually called
// after parsing the request body to determine the model and invoke the routing logic.
SetBackend(backend *filterapi.Backend)
// RecordTokenUsage records token usage metrics.
RecordTokenUsage(ctx context.Context, inputTokens, outputTokens, totalTokens uint32, extraAttrs ...attribute.KeyValue)
// RecordRequestCompletion records latency metrics for the entire request
RecordRequestCompletion(ctx context.Context, success bool, extraAttrs ...attribute.KeyValue)
// RecordTokenLatency records latency metrics for token generation.
RecordTokenLatency(ctx context.Context, tokens uint32, extraAttrs ...attribute.KeyValue)
}
ChatCompletionMetrics is the interface for the chat completion AI Gateway metrics.
type NewCustomChatCompletionMetricsFn ¶ added in v0.2.0
type NewCustomChatCompletionMetricsFn func(meter metric.Meter) ChatCompletionMetrics
NewCustomChatCompletionMetricsFn is the function to create a custom chat completion AI Gateway metrics.
var NewCustomChatCompletionMetrics NewCustomChatCompletionMetricsFn
NewCustomChatCompletionMetrics is the function to create a custom chat completion AI Gateway metrics over the default metrics. This is nil by default and can be set by the custom build of external processor.
type NewCustomRouterFn ¶
NewCustomRouterFn is the function signature for NewCustomRouter.
It accepts the exptproc config passed to the AI Gateway filter and returns a Router. This is called when the new configuration is loaded.
The defaultRouter can be used to delegate the calculation to the default router implementation.
var NewCustomRouter NewCustomRouterFn
NewCustomRouter is the function to create a custom router over the default router. This is nil by default and can be set by the custom build of external processor.
type Router ¶
type Router interface {
// Calculate determines the route to route to based on the request headers.
//
// The request headers include the populated [filterapi.Config.ModelNameHeaderKey]
// with the parsed model name based on the [filterapi.Config] given to the NewCustomRouterFn.
//
// Returns the selected route rule name and the error if any.
Calculate(requestHeaders map[string]string) (route filterapi.RouteRuleName, err error)
}
Router is the interface for the router.
Router must be goroutine-safe as it is shared across multiple requests.
Source Files