Documentation
¶
Overview ¶
Package llmutil provides common utilities for LLM container testing.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type APIResponse ¶
type APIResponse[T any] struct { // Objects is the list of JSON objects in the response. Objects []*T // Metrics contains HTTP response metrics. Metrics ResponseMetrics }
APIResponse represents a response from an LLM API.
func MakeAPIResponse ¶
func MakeAPIResponse[T any](rawResponse []byte) (*APIResponse[T], error)
MakeAPIResponse decodes a raw response from an instrumented HTTP request.
func (*APIResponse[T]) Obj ¶
func (ar *APIResponse[T]) Obj() (*T, error)
Obj returns the first object in the response, if there is a singular object in the response.
type DockerServer ¶
type DockerServer struct {
Container *dockerutil.Container
Logger testutil.Logger
Port int
ClientImage string
}
DockerServer implements an LLM server interface via a local Docker container.
func (*DockerServer) InstrumentedRequest ¶
func (ds *DockerServer) InstrumentedRequest(ctx context.Context, argvFn func(hostPort string) []string) ([]byte, error)
InstrumentedRequest performs an instrumented HTTP request against the server.
type ResponseMetrics ¶
type ResponseMetrics struct {
// ProgramStarted is the time when the program started.
ProgramStarted time.Time `json:"program_started"`
// RequestSent is the time when the HTTP request was sent.
RequestSent time.Time `json:"request_sent"`
// ResponseReceived is the time when the HTTP response headers were received.
ResponseReceived time.Time `json:"response_received"`
// FirstByteRead is the time when the first HTTP response body byte was read.
FirstByteRead time.Time `json:"first_byte_read"`
// LastByteRead is the time when the last HTTP response body byte was read.
LastByteRead time.Time `json:"last_byte_read"`
}
ResponseMetrics are HTTP request metrics from an LLM API query.
func (*ResponseMetrics) TimeToFirstByte ¶
func (rm *ResponseMetrics) TimeToFirstByte() time.Duration
TimeToFirstByte returns the duration it took between the request being sent and the first byte of the response being read.
func (*ResponseMetrics) TimeToLastByte ¶
func (rm *ResponseMetrics) TimeToLastByte() time.Duration
TimeToLastByte returns the duration it took between the request being sent and the last byte of the response being read.
type Server ¶
type Server interface {
// InstrumentedRequest performs an instrumented HTTP request against the
// server, using the client image.
InstrumentedRequest(ctx context.Context, argvFn func(hostPort string) []string) ([]byte, error)
// Logs retrieves logs from the server.
Logs(ctx context.Context) (string, error)
}
Server performs requests against a generic LLM server.
Source Files