Documentation
¶
Index ¶
- func ParseUsage(body []byte, requestURL string) (input, output int64, ok bool)
- func WithCancellationChecker(ctx context.Context, checker asyncapi.CancellationChecker) context.Context
- func Worker(consumeCtx, requestCtx context.Context, ...)
- func WorkerWithGate(consumeCtx, requestCtx context.Context, ...)
- type HTTPInferenceClient
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func ParseUsage ¶ added in v0.9.1
ParseUsage extracts the OpenAI usage object from a response body. ok is false when requestURL does not point at an OpenAI-style completions endpoint, when the body is not valid JSON, or when it carries no usage object; such bodies are silently ignored (best-effort parse on a hot path). Negative token counts are clamped to 0 so they can never panic a counter Add.
func WithCancellationChecker ¶
func WithCancellationChecker(ctx context.Context, checker asyncapi.CancellationChecker) context.Context
WithCancellationChecker attaches a request cancellation checker to the worker context.
func Worker ¶
func Worker(consumeCtx, requestCtx context.Context, characteristics pipeline.Characteristics, client asyncapi.InferenceClient, requestChannel chan pipeline.EmbelishedRequestMessage, retryChannel chan pipeline.RetryMessage, resultChannel chan asyncapi.ResultMessage, requestTimeout time.Duration, transforms *transform.Chain)
func WorkerWithGate ¶
func WorkerWithGate(consumeCtx, requestCtx context.Context, characteristics pipeline.Characteristics, client asyncapi.InferenceClient, requestChannel chan pipeline.EmbelishedRequestMessage, retryChannel chan pipeline.RetryMessage, resultChannel chan asyncapi.ResultMessage, requestTimeout time.Duration, transforms *transform.Chain, poolGate pipeline.Gate)
Types ¶
type HTTPInferenceClient ¶
type HTTPInferenceClient struct {
// contains filtered or unexported fields
}
HTTPInferenceClient is the default HTTP implementation of InferenceClient.
func NewHTTPInferenceClient ¶
func NewHTTPInferenceClient(client *http.Client) *HTTPInferenceClient
NewHTTPInferenceClient creates a new HTTPInferenceClient with the given HTTP client.
func (*HTTPInferenceClient) SendRequest ¶
func (h *HTTPInferenceClient) SendRequest(ctx context.Context, url string, headers map[string]string, payload []byte) (*asyncapi.InferenceResponse, error)
SendRequest implements InferenceClient for HTTP-based inference requests.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package transform defines the request body-transform extension point: a typed plugin capability that lets providers rewrite the outgoing HTTP body at dispatch time (e.g.
|
Package transform defines the request body-transform extension point: a typed plugin capability that lets providers rewrite the outgoing HTTP body at dispatch time (e.g. |
|
gcsmultipart
Package gcsmultipart provides a RequestTransform plugin that rewrites the OpenAI-style JSON body into multipart/form-data for multi-modal inference endpoints (e.g.
|
Package gcsmultipart provides a RequestTransform plugin that rewrites the OpenAI-style JSON body into multipart/form-data for multi-modal inference endpoints (e.g. |
Click to show internal directories.
Click to hide internal directories.