Documentation
¶
Index ¶
Constants ¶
View Source
const (
// Name is the backend name.
Name = "sglang"
)
Variables ¶
Functions ¶
func GetContextLength ¶
func GetContextLength(modelCfg types.ModelConfig, backendCfg *inference.BackendConfiguration) *int32
GetContextLength returns the context length (context size) from backend config or model config. Backend config takes precedence over model config (runtime configuration). Returns nil if neither is specified (SGLang will auto-derive from model).
func New ¶
func New(log logging.Logger, modelManager *models.Manager, serverLog logging.Logger, conf *Config, customPythonPath string) (inference.Backend, error)
New creates a new SGLang-based backend. customPythonPath is an optional path to a custom python3 binary; if empty, the default path is used.
Types ¶
type Config ¶
type Config struct {
// Args are the base arguments that are always included.
Args []string
}
Config is the configuration for the SGLang backend.
func NewDefaultSGLangConfig ¶
func NewDefaultSGLangConfig() *Config
NewDefaultSGLangConfig creates a new SGLangConfig with default values.
func (*Config) GetArgs ¶
func (c *Config) GetArgs(bundle types.ModelBundle, socket string, mode inference.BackendMode, config *inference.BackendConfiguration) ([]string, error)
GetArgs implements BackendConfig.GetArgs.
Source Files
Click to show internal directories.
Click to hide internal directories.