Documentation
¶
Overview ¶
Package sglang provides an SGLang API client.
Index ¶
- Constants
- type MetaInfo
- type ModelLoadStats
- type Prompt
- type Response
- func (r *Response) Done() bool
- func (r *Response) E2ELatency() float64
- func (r *Response) NumTokens() int
- func (r *Response) OutputTokensPerSecond() float64
- func (r *Response) String() string
- func (r *Response) Text() string
- func (r *Response) TimeToFirstToken() time.Duration
- func (r *Response) TimeToLastToken() time.Duration
- type SGLang
- func (llm *SGLang) Prompt(ctx context.Context, prompt *Prompt) (*Response, error)
- func (llm *SGLang) PromptUntil(ctx context.Context, prompt *Prompt, ...) (*Response, error)
- func (llm *SGLang) WaitUntilServing(ctx context.Context) error
- func (llm *SGLang) WarmModel(ctx context.Context) (*ModelLoadStats, error)
Constants ¶
const (
// Port is the port used by the sglang server.
Port = 30000
)
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type MetaInfo ¶
type MetaInfo struct {
// E2ELatency only exists in the last token.
E2ELatency float64 `json:"e2e_latency"`
}
MetaInfo contains the meta information about the response. Every token has a meta info.
type ModelLoadStats ¶
type ModelLoadStats struct {
// ClientReportedDuration is the duration to load the model as perceived
// by the client, measured by HTTP client metrics.
ClientReportedDuration time.Duration
}
ModelLoadStats holds metrics about the model loading process.
type Prompt ¶
type Prompt struct {
// Text is the prompt string.
// Common leading whitespace will be removed.
Text string
// Options maps parameter names to JSON-compatible values.
Options map[string]any
// contains filtered or unexported fields
}
Prompt is an sglang prompt.
func ZeroTemperaturePrompt ¶
ZeroTemperaturePrompt returns a Prompt with the given text and an initial temperature setting of zero. This setting allows for consistent settings.
func (*Prompt) AddImage ¶
AddImage attaches an image to the prompt. Returns itself for chainability.
func (*Prompt) CleanQuery ¶
CleanQuery removes common whitespace from query lines, and all leading/ending whitespace-only lines. It is useful to be able to specify query string as indented strings without breaking visual continuity in Go code. For example (where dots are spaces):
"""\n ..The Quick Brown Fox\n ..Jumps Over\n ....The Lazy Dog\n ."""
becomes: Jumps Over\n ..The Lazy Dog"""
func (*Prompt) RaiseTemperature ¶
func (p *Prompt) RaiseTemperature()
RaiseTemperature increases the "temperature" option of the model, if any.
func (*Prompt) SetMaxNewTokens ¶
SetMaxNewTokens sets the "max_new_tokens" option of the prompt to the given value.
func (*Prompt) SetTemperature ¶
SetTemperature sets the "temperature" option of the prompt to the given value.
func (*Prompt) WithHotterModel ¶
WithHotterModel returns a copy of this prompt with the same model having a higher temperature.
type Response ¶
type Response struct {
// contains filtered or unexported fields
}
Response represents a response to a query from SGLang.
func (*Response) E2ELatency ¶
E2ELatency returns the elapsed time between when start_time was recorded and the current moment in seconds. See https://github.com/sgl-project/sglang/blob/4a2768a86b2905b9b7f19d415261b9d4af639e19/sgl-router/src/routers/grpc/regular/streaming.rs#L904
func (*Response) OutputTokensPerSecond ¶
OutputTokensPerSecond computes the average number of output tokens generated per second.
func (*Response) TimeToFirstToken ¶
TimeToFirstToken returns the time it took between the request starting and the first token being received by the client.
func (*Response) TimeToLastToken ¶
TimeToLastToken returns the time it took between the request starting and the last token being received by the client.
type SGLang ¶
type SGLang struct {
// contains filtered or unexported fields
}
SGLang is an sglang client.
func New ¶
New starts a new SGLang server in the given container, then waits for it to serve and returns the client.
func NewDocker ¶
func NewDocker(ctx context.Context, cont *dockerutil.Container, logger testutil.Logger) (*SGLang, error)
NewDocker returns a new SGLang client talking to an SGLang server that runs in a local Docker container.
func (*SGLang) PromptUntil ¶
func (llm *SGLang) PromptUntil(ctx context.Context, prompt *Prompt, iterate func(*Prompt, *Response) (*Prompt, error)) (*Response, error)
PromptUntil repeatedly issues a prompt until `iterate` returns a nil error. `iterate` may optionally return an updated `Prompt` which will be used to follow up. This is useful to work around the flakiness of LLMs in tests.
func (*SGLang) WaitUntilServing ¶
WaitUntilServing waits until sglang is serving, or the context expires.
func (*SGLang) WarmModel ¶
func (llm *SGLang) WarmModel(ctx context.Context) (*ModelLoadStats, error)
WarmModel pre-warms a model in memory and keeps it warm for `keepWarmFor`. If `unloadFirst` is true, another model will be loaded before loading the requested model. This ensures that the model was loaded from a cold state.
Source Files